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'' That man, I think, has had a liberal education wl#o 
has been so trained in youth that his body is the ready ser- 
vant of his will, and does with ease and pleasure all the work 
that, as a mechanism, it is capable of ; whose intellect is a 
clear, cold, logic engine, with all its parts of equal strength, 
and in smooth working order ; ready, like a steam engine, 
to be turned to any kind of work, and spin the gossamers 
as well as forge the anchors of the mind ; whose mind is 
stored with a knowledge of the great and fundamental truths 
of Nature and of the laws of her operations ; one #ho, no 
stunted ascetic, is full of life and fire, but whose passions are 
trained to come to heel by a vigorous will, the servant of 
a tender conscience ; who has learned to love all beauty, 
whether of Nature or of art, to hate all vileness, and to 
respect others as himself. 

Such an one and no other, I conceive, thas had a liberal 
education. — Huxley, Science and Education, p. 86. 
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In the first paper in this little book, delivered as 
a lecture at the Royal Society of Arts in January 
1911 (and published in a first edition by Hugh 
Rees), I put forward a plea, which received the 
powerful support of Lord Cromer, for a Royal 
Commfssion to inquire into the working of the 
examination system. The paper was discussed 
in a number of leading articles in the press. I 
cannot say what influence the paper itself, or 
Lord Cromer’s^ views, or pubhc comments, had 
on the Government. At any rate, in March 1912, 
a Royal Commission, presided over by Lord 
MacDonnell, was appointed with the following 
reference : i 

To inquire into and report on the methods 
of making appointments to and promotions 
in the Civil Service, including the Diplomatic 
and Consular Services, and the legal depart- 
ments ; 

To investigate the working and efficiency 
of the system of competitive examination 
ix • 
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for such appointments, and to make recom- 
mendations for any alterations or improve- 
ments in that system which may appear to 
be advisable; and 

To consider whether the existing scheme 
of organisation meets the requirements of 
the Public Service, and to suggest any modifi- 
cations which may be needed therein. 

In April 1914 (after three Reports of a formal 
nature accompanying the issue of volumes of 
evidence) the Commission published its Fourth 
Report (with a Minority Report) in a volume 
extending to over 150 folio pages.* The last 
paragraph of the Majority Report reads as fgllows ; 
“ In submitting this Report on the first part of 
our reference we desire," etc. — here follows a 
well-deserved tribute to the work of the Secre- 
tary. 

Now the Report, as a matter* of fact, deals 
not only with the first, but with the third part 
of the reference, and contains important sugges- 
tions for the reorganisation of the Civil Service. 
But with the second part of the reference — the 
working and efficiency of the examination systein 
— the Commission neither profess to have dealt 
nor have they dealt, except in a most fragmentary 
fashion, and it is important to bring out this fact 
lest it be supposed that the work which the 
Commission were asked to do in regard to examina- 

* Royal Commiwioii 00 the Civil Service [Cd. 7338]. 



INTRODUCTION 

tions has, in fact, been carried out. I quote in 
Note A, pp. 80-88 below, the main passages of 
the Report dealing with the working of the 
examination system. 

I have been unable to discover in the whole of 
the Fourth Report any contribution to our 
knowledge of the working of that system of 
examinations, or any suggestion as to its improve- 
ment. The Commission say that they propose 
“ in due place ” [see Note A, p. 81] to examine 
the defects attaching to the system, and suggest 
that they are susceptible of " some mitigation.” 
But the examination of these defects is reserved 
for a further Report, of which, perhaps owing 
to th^war, there has been no sign. In their 
formal Recommendations the Commission say 
that examinations should be directed to testing, 
inter alia, “ the formation of character ” * ; but 
they give no indication of how this difficult test 
is to be carried*out. I have no desire to criticise 
the Royal Commission in this matter ; I wish to 
point out that they have furnished no matter 
for criticism. It would be convenient to know 
whether the Commission have any intention of 
completing the investigation with which they 
were entrusted. A large body admirably repre- 
senting many important interests, but not par- 
ticularly well acquainted with the examination 
system, may well have recoiled before an expert 
and ungrateful task of this kind, and have rested 

^ See Report, p. loi, quoted in Note A, p. 88. 
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content with the weighty conclusions they have 
reached in regard to other matters. But it is 
a fact that ignorance of the real meaning and 
working of our examinations remains in 1917 
what it was in 1911 ; and the plea for a com- 
prehensive scientific investigation holds as good 
now as it did then. 

It would, however, be wrong not to point out 
the very fruitful result to which one of the sug- 
gestions of the Majority Report of the Commis- 
sion * has led in the shape of the Report of Mr. 
Stanley Leathes' Committee on Civil Service 
Examinations. If the recommendations of the 
Committee are adopted, they wiU mark a new 
era in our system of competitive examjpations 
by the importance they will give to that part of 
examinations for which cramming is impossible, 
and they will go far to carry out certain sugges- 
tions which I have made in regard to this subject.* 

The object of the second paper*in this volume, 
originally published in The Times Educational 
Supplement of May loth, 1917, though it covers 
somewhat the same ground as the first, is different.* 


‘ See Majority Report, § 50, quoted on p. 87, below, 
s For details in regard to the personnel and reference of 
the Committee and a summary of their recommendations, 
see Note B, p. 89 below, 

® The paper, as actually written, though composed for 
another purpose, was submitted to Mr. Stanley Leathes' 
Committee at the request of one of the members ; it was 
not printed quite in extenso in the Educational Supplement, 
I have made a few modiheations of it for its republication. 

# xii 
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It is intended to give a more general perspective 
of the subject as a whole, though in the last para- 
graph I have not been able to resist the tempta- 
tion to appeal once more to examining bodies 
to tell us in clear and unmistakable terms the 
meaning of their various examination certificates. 
The paper may be regarded as a series of chapter 
headings for a larger work which I have had in 
mind but which I have not had the leisure to write. 
The subject, important in 1911, is even more 
important now Mr. Fisher’s remarkable speeches 
as President of the Board of Education, and his 
Education Bill of this year, seem to show that the 
Government are in earnest about educational 
reconstruction, that the long campaign of enlight- 
enment and propaganda in which Mr. M. E. 
Sadler has taken the chief part is at last about 
to bear fruit. The working classes and their 
leaders, whose share in the government of the 
country is rapidly becoming of increasing — I was 
almost saying predominant — importance, are no 
less in earnest. But in dealing with examina- 
tions there is still timidity of thought and action 
that may go far to wreck our new-born hopes. 

* * Let it not be said that examinations do not 
matter. Every teacher, every administrative 
educational body, knows that, whatever be the 
schemes and curricula, examinations do effec- 
tively control the class-rooms of our Secondary 
Schools, and that they control to a considerable 
extent our Universities also ; and no educational 

xiii • 
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reform which leaves our examination system as it 
is, will or can be a real reform. Professor John 
Burnet, the distinguished philosopher and Dean 
of the Faculty of Arts at St. Andrews, has written 
an interesting book on Higher Education and the 
War} There are few pages in it, after the initial 
chapters on " Kultur and Humanism ” that do 
not refer to examinations, directly or indirectly. 
Yet, speaking (on page 195) of classical teaching 
Professor Burnet declares that ; “ The written 
part of the examination can only deal successfully 
with that part of it which is destined to be for- 
gotten as soon as it has served its purpose, and it 
can tell us next to nothing of what is to remain 
as a possession for ever.” I do not disagree with 
Professor Burnet. On the contrary, I wish he 
had travelled with me further. And in this con- 
nection I want to plead for the retention of our 
English word “ culture,” which is in so great a 
danger of being ostracised through the recent 
degeneration of its German cousin Kultur} I 
want the word especially because it enables me 
to summarise briefly my view of examinations 
in relation to education as a whole. Using the 
word as I have myself used it in the text, or Ssf- 
Professor A. N. Whitehead has recently defined it * 


^ Macmillan 8c Co., 1917. 

* See Note C, p. 93, below. 

^ In his original and stimulating series of essays entitled, 
Thfi Organisation of Thought (Williams 8c Norgate, 19x7), 
p. S and passim. 
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— “Activity of thought, and receptiveness to 
beauty, and humane feeling “ — it exactly coven 
that vital part of education which cannot he tested 
by the ordinary written examination. Examina- 
tions test, and can test, that part of a man's 
(or a woman’s) education in which it is useful 
and necessary to think and act more or less on 
a jnodel, and which I have called education for 
technical efficiency. Culture is that part of 
education which is meaningless if it is not sensi- 
tive and individual. 

It is convenient to divide education into educa- 
tion for technical efficiency and education for 
culture, remembering, however, that efficiency in 
its widest sense implies culture also. The social 
fitness of an individual depends not on one of 
these factors alone but on both, and both must 
be considered in devising our new schemes of edu- 
cation. My complaint of reformers like Professor 
Burnet is that Hheir criticism of the much-abused 
examination system leaves it as supremely un- 
reformed as ever. And I urge two correlative 
reforms : first, that examinations should be made 
giuch more real and drastic tests of technical 
efficiency than they are at present ; and, secondly, 
that a large portion of the educational field should 
be entirely freed from their influence, hi what 
foUows I have given my reasons for regarding 
many of our present exami|ations as insufficiently 
searching in matters recognised as of the first 
importance,; but I may give an example here of 
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the use which should be made, in my judgment, 
of that portion of the school time-table which I 
hope some day to see freed from examination 
control altogether. I should like to see every 
boy and girl who intends to follow a scientific 
career obliged to attend classes in literature and 
history with no examination on these subjects, 
the teacher being forced either to interest his 
pupils or to suffer from their boredom, without 
any examination whip at his command, and 
I should like to see every boy or girl who intends 
to follow an " Arts " career obliged to attend 
classes in natural science of the same kind. What 
a revolution we should have in the teaching of 
these non-examination classes I What a Messing 
for the good teacher ; what a stimulus for the 
poor one ! I desire here to protest emphatically 
against the idea of Mr. R. W. Livingstone, ex- 
pressed in his recent book,* that^science is “un- 
imaginative." It is quite true that elementary 
science can be extraordinarily dull and inhuman. 
So (as he sees) can elementary history, or Latin 
accidence ; they are so in the hands of teachers 
who have neither historical training nor phila-, 
sophic insight. The same holds good for 
science, and one is bound to admit that much 
of the teaching of “ science " has been in the 

^ K. W. Livingatone — A Defence of Classical Education, 
1916. [Some of the attacks on Mr. Livingstone's twok, 
which seems to me in many ways admirable, are notthatified 
by the text, in which he claims for science a great diare la 
education (cf. pp. 51 et seig.)}. 
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past all that Mr. Livingstone depicts it — un- 
critical, unhistorical, unscientific, unaware that 
our scientific description of nature is constantly 
changing with the progress of science, or that 
the achievements of science are marked by the 
individualities of great men as certainly as the 
achievements of literature. Even in mathe- 
matics, the most compelling of all sciences, the 
presentation of truths is not a unique presenta- 
tion which imposes itself on all alike ; there are 
many presentations possible, and our traditional 
methods, now so largely changing, are character- 
ised by the individualities of the great mathe- 
maticians of the past. While it is true that in 
literattUre and in art there can be no compulsion 
of the individual to admire or accept anything 
(although pro tanto, we must be content to be 
regarded as barbarians if we refuse our admira- 
tion to certain^ of the " greatest ” men) ; and, 
on the contrary, in science the compulsion in 
regard to some fundamental truths is forced on 
us by our physical environment, yet there is a 
great asthetic and individual side in science, of 
which, from pure ignorance, many literary men 
fail to appreciate the existence. The terms “ a 
beautiful theorem," "a beautiful experiment," 
"a beautiful theory," correspond to beauty as 
real to those who feel it as the beauty of a poem 
or a picture. The " wonders of science " com- 
monly brought forward for our admiration are 
small things compared with the evolution of the 

xvii 
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Newtonian theory of gravitation from the obser- 
vations of Kepler, or of the evolution of Ampdre’s 
theory of electro-dynamics from a few simple 
experiments of Oersted. But the culture of 
science, like the culture of art, is not for the 
examination-room. 

If the problems of examination touch the 
general problems of education at every point they 
nevertheless require separate investigation and 
treatment ; and it is for this investigation that I 
plead, and to which I hope that this little book 
may be regarded as a contribution. Will the 
new Secondary Schools Examination Council 
created by Mr. Fisher tread the old paths ? Or 
will they avail themselves of a really gr^t and 
new opportimity ? ^ 

August 1917. r. J- ti. 

Owing to absence abroad the author has been unable 
to see this book through the press, ai^ he desires to ex- 
press his sincere thanks to Professor T. P. Nunn and to 
Mr. Arthur Watson, who have most kindly revised the proofs 
for him, and also to Professor F. Y. Edgeworth for his help 
in the revision of Note E, 

[The two following essays are based in part not only on prt-^ 
vious publications by the writer, but also on a joint article 
on “ Examinations " by Mr. Arthur Watson and himself in the 
eleventh edition of the Encfclopadia Britannica, which con- 
tains a fairly extensive bibliography of the subject. The 
author desires, further, to refer his readers to the article on 
Examinations signed by Prof. A. N. Whitehead as Chair- 
man of the relevant committee of the '' Education Reform 
Council/' and published in the valuable collection of Reports 
entitled Education Reform (P. S. King & Son, 1917),] 

•xviu 



EXAMINATIONS AND THEIR 
RELATION TO CULTURE 
AND EFFICIENCY 

I 

EXAMINATIONS IN THEIR BEARING 
ON NATIONAL EFFICIENCY 

{A lecture delivered at the Royal Society of Arts in 
January 1911.) 

May I begin with a personal explanation ? What 
I have to say to-night is said on my own behalf 
alone, and no one else in the University which I 
have the honour to serve must be regarded as 
committed in any way to agreement either with 
my main thesis or with any incidental opinions I 
may express. 

To avoid all possible misunderstanding I shall 
^^tate my position at the outset. I do not desire 
to appear as counsel either for the plaintiff or 
for the defendant in the great controversy on 
examinations which has raged intermittently 
for more than half a century in this country, as 
in other countries, but rather as one of the public, 
deeply interested in the issues, and trying to see 
clearly thjpugh the heat and dust of the conflict. 
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It is often said, and generally accepted, that 
the whole of English education is controlled by 
the examination system. It is well, however, in 
dealing with educational questions, to keep some- 
where in the back of one’s mind the fact that 
there is another kind of education than the educa- 
tion of the schools, the education that Mr. Samuel 
Weller’s father gave him when he let him have 
the run of the streets and make shift for himself ; 
well to remember that there is one branch of the 
national activity, not the least successful or 
important — I mean its commerce — which up to 
the present has been but little axiected by examina- 
tion-ideals ; and that our agriculture and indus- 
tries have been artected by them only to af minor 
extent. Our business, our agriculture, our indus- 
tries, have to a large extent “ shifted for them- 
selves,” I will not say without the help of school 
teaching, but very largely with^qut the help of 
examinations. 

But the public demands that persons on whose 
services it rehes, and for whose failures it cannot 
be compensated, as by a business man who fails 
to fulfil his contract — that these should produo?,, 
some certificate of competency based on an 
examination, and often on a series of examina- 
tions beginning in childhood and prolonged into 
early manhood and beyond. Teachers, lawyers, 
doctors, dentists, engineers, architects, and the 
civil, naval, and miUtary servants of the Crown, 
on whom the orgat^sation and defonce of the 

a 
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Empire rest, must pass examination tests, and 
in many cases their whole career is determined by 
such tests. 

It is, therefore, a matter of national and 
imperial concern that this method of test should 
be wise in its design, certain and not capricious 
in its working, and, above all, that in its opera- 
tions it should not damage the human material 
on which it sets a hall-mark. 

Can it be said of English examinations at the 
present day that they fulfil these requirements ? 
Are they wisely designed ? Do they work with 
certainty and accuracy ? Do they tend, on the 
whole, to improve the boys and girls, the men 
anck w^men, whose education, whose lives and 
careers they so largely control ? Are they really 
serving in the best possible way the national and 
imperial purposes they are supposed to serve ? 
If these questions were put to a Referendum, I 
have little doubt of the reply. 

While it may be conceded that there are many 
judicious examiners, and examining bodies of 
unquestioned wisdom, the body educational as a 
jtthole is suffering from the aches and pains of 
examination. Those who say, " Get rid of 
examinations altogether,” offer us nothing in 
their place ; they do not realise the situation. 

I spoke just now as if examinations affected us 
only from without ; but have not examinations 
become, as it were, an artificial nervous system 
of our edvjjiation, of which every movement is 
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controlled either by their stimulus or by their 
power of inhibition, so that extirpation is an 
impossible task ? If we cannot remove this tyrant 
that governs education from within, can- 
not we either reduce its powers or force it into 
beneficence ? Quixotic as it may seem, I believe 
the task not to be impossible. 

The task demands inquiry — and by inquiry I 
mean scientific investigation — patience, and per- 
haps some drastic reform later. My object to- 
night is to suggest, first of all, the outlines of an 
inquiry into our examination system, its design, 
its methods, and its results ; and I shall use the 
word “ examination " to mean a systematic test 
of knowledge, and of either special or ^en»ral 
capacity or fitness, carried out under the authority 
of some public body. 

First of all, the design of examinations. We 
cannot understand that design without one word 
as to their history and their original function. 
The origin of Western examinations — I say nothing 
of those Chinese examinations which date back 
three or four thousand years — is sufficiently well 
known to the historians ; they are derived fro^ji^ 
the examinations of - the mediaeval universities. 
Those tmiversities were corporations of teachers 
and students, analogous to the trade and craft 
gilds of the time. In the gilds there were appren- 
tices, companions, and masters. No one was 
admitted to mastership until he had served his 
apprenticeship, nor, as a rule, un^U he had 
' 4 
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shown that he could accomplish a piece of work 
to the satisfaction of the gild. The object of the 
University was to teach ; its scholars, bachelors, 
or pupil teachers, and masters or doctors, corre- 
sponded roughly to the three classes established 
by the gilds, and the master or doctor (two terms 
at first equivalent) was one who. having served 
hi§ apprenticeship and passed a definite technical 
test, had received the Ucentia docendi — the per- 
mission to teach. The candidate was required 
to teach in public at the examination precisely as 
he would be required to teach later in the schools. 
The examination fulfilled exactly the technical 
purpose for which it was intended. We are apt 
to j j i ittk of the schoolmen and all their ways with 
a contempt inherited from the days of Bacon and 
Descartes ; but in discussing examinations of this 
kind, that distinguished historian of education, the 
late Professor Paulsen, of Berlin, asserts that they 
were well adapted to increase a student's alert- 
ness, his power of comprehending new ideas, and 
his ability quickly and surely to assimilate them 
to his own ; and that they did more to enable 
students to grasp subjects than the mute and 
^solitary reviewing arid cramming for our modem 
examinations can possibly do.‘ The limitations 

^ Germany has frequently in the past been held up to 
English reformers as a model country in which no examina- 
tions similar to those of Great Britain are held. The writers 
have in mind chiefly the question of University examinations, 
and are unaware of the elaborate system of State examina- 
tions which guard every professional career in Germauay 
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of the test were the limitations of the educational 
and philosophic ideals of the time. At their best, 
as I say, they did exactly what they were required 
to do. At their worst, says Canon Rashdall, they 
were less of a farce than certain English Univer- 
sity pass examinations almost within the memory 
of persons now living, ‘ 

To trace the history of Western examinations 
from their origin to our own time we should need 
to follow up the ever-changing course of educa- 
tional ideals with the development of modem 
societies through seven centuries, and the manifold 
actions and reactions during all that period be- 
tween education ideals and examination ideals. 
For such a task we have not even the materials 
prepared ; but, remembering that the original 
examination was a test of technical efficiency, 
excellently suited to its purpose, we can see how 
examinations have taken on two, and perhaps 
three, entirely new functions, anfi with far-reach- 
ing effects both on examinations and on education. 

The first new function— the use of examinations 
to distinguish between candidates of different 
merit — was an early and almost inevitable develop- 

except that of a University teacher, and are equally unaware 
of the severe system of school examinations, condemned by 
Paulsen, and dealt with by Prof. Burnet in his book on 
Higher Education and the War, See F. Paulsen, The German 
Universities, trans. by F. Thilly and W. Elwang, p. 25, and 
Geschichte des gelehrten Unterrichts, vol. ii., pp. 368 et seq, 

‘ The Universities of Europe in the Middle Ages, by Hast- 
ings Rashdall, vol. i., p. 4^0 (Clarendon Press, 1695). 

6 
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ment ; and yet we can see every day how the 
competitive idea leads us to forget the distinction 
between the efficiency of an examination and its 
difficulty. It is constantly said by examination- 
enthusiasts: "This examination is a superior 
examination to that, because it is harder to pass.” 
Should we not rather ask : "Is the man who 
passes it a more useful member of society ? ” In 
examinations professedly technical, like the 
original examinations, this question is of the 
utmost importance. Here the main question is, 
surely, not which candidate can do the most 
difficult things, solve the most Chinese of puzzles, 
remember the most unrememberable of formulae, 
but whether the examination is helping to turn 
oBT ffie best teachers, doctors, engineers, indus- 
trial botanists, industrial chemists. In other 
examinations, like the highest Civil Service ex- 
aminations, ' the element of pure difficulty, on 
which I shall„ have a word to say later, may 
appear in another light. 

The second new function of examinations is a 
later development, and in some ways a character- 
istically, though not exclusively, English develop- 
•>ment — I mean their, use to test not so much the 
efficiency of candidates as that of their schools. 
The external examination of schools and of school- 
children, begun by the College of Preceptors in 
1853, and now carried out on an immense scale 
by the College and by other bodies, mainly Uni- 
versity bodies, all over the United Kingdom, had 

7 
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primarily for its object to distinguish between the 
sheep and the goats of the private schools. The 
examination, as thus used, was organised pri- 
marily as a technical test, good or bad, not of the 
taught, but of the teacher. But are not the real 
' interests of the taught, which it was originally in- 
tended to serve, sometimes, or perhaps often, lost 
sight of in the process ? Here, again, the difficulty 
of an examination is apt to be confused with its 
efficiency. 

There is a third function of examinations, not 
easy to distinguish altogether from the original 
function of testing efficiency — I mean their use 
to test what is called “ general culture.” In deal- 
ing with this question we find a certain amount 
of common agreement as to what is the* 6?st 
policy, combined with a powerlessness to carry 
out that policy in the face of our present organisa- 
tion of examination and education, and our 
refusal in education to remember*the wisdom of 
leaving some things the time and opportunity to 
” shift for themselves.” 

It will, I think, be generally admitted that 
culture is something more than the knowledge of 
the best that has been thought and said in the 
world, according to the definition of Arnold, unless 
knowledge is used with a far wider connotation 
than is possible in the examination-room — some- 
thing perhaps more personal and sensitive than 
the ” enlargement ” or ” Uluminative reason " 
which John Henry Newman contrasts with know- 

8 
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ledge.* I shall enter into no rivalry of epigram 
with these great masters of style ; but I think, on 
cross-questioning, we should find that the cultured 
man is the man who is sensitive and responsive 
over a large field of knowledge regarded as of 
value by the commimity. On further cross- 
questioning we should again find it generally 
admitted that that sensitiveness and responsive- 
ness are apt to suffer injury from the intellectual 
repression required for the examination syllabus. 
At every step the delicate feelers of the mind are 
paralysed by the suggestion : “I am wasting my 
time in going further; that won't be asked." 
It may be held, and I should agree, that culture 
is as ind ividual a thing as conscience ; that culture 
m5f^e killed, that it cannot be caught, by 
examinations. Yet teachers who realise all this, 
who think examinations in their subject mis- 
chievous rather than helpful, implore the authori- 
ties to include It in every possible examination 
syllabus. Why ? Because, under the present 
regime, a subject that is not examined in is likely 
to disappear speedily from our teaching curricula. 
That point is sufficiently illustrated by the con- 
ftroversy on examinations in Greek. 

We have thus seen that examinations are nowa- 
days conducted with four distinct primary 
objects : — 

(i) To test the efficiency of a person for the 
practice of his technical calling or profession. 

*■ See Note C, p. 93, below. 

9 
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(2) To arrange candidates in an order of merit. 

(3) To test the efficiency of schools. 

(4) To promote general culture, an object for 
which many persons regard them as totally un- 
suited. 

I am quite aware of the complexities of the 
issues raised by this simple classification . A recen t 
writer, dealing with the same problem from a 
different point of view, has made out a list ‘of 
seventeen qualities which an examination can 
test, and thinks his list might be extended still 
further. My classification deals only with groups 
of such qualities, and the grouping will vary with 
the specific object of an examination. I think 
it quite possible that I should find it difficult 
to conciliate all my critics, and most of alFtHOse 
who are partial to the examinations in Class (4) 
of my list. I am however willing, as I said 
before, to concede that any particular examina- 
tion, and any particular exanii*iing body, may 
have all the virtues claimed for them ; but I 
would ask of those concerned if it is not a fact 
that if we took the examination-papers set at 
many examinations, and inspected them in detail, 
we should find it an impossible task to decide, frotn^, 
internal evidence, with what object, or with what 
combination of objects, those examinations are 
organised. Individual examiners are not asked 
to bear in mind, nor is it, perhaps, always desired 
that they should bear in mind, the careers which 
the examinees propose to follow ; they are asked, 
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not to examine with any particular aim in view, 
but to examine " in their subject,” and this, I 
suppose, is what is meant by the opponents of 
examinations when they rail against examina- 
tion for examination’s sake. Thus I am inclined 
to think that the most ardent advocate of the 
examination system would be prepared to admit 
that the object of many examinations held nowa- 
days is indeterminate. It can scarcely be ex- 
pected, therefore, that the results will be in all 
eases precise. 

Before we can estimate in detail the results of 
examinations we must say something of their 
methods. Examination-tests may be classified in 
t wo wa ys : first, according to what may be called 
tH?!?extemal character, under which heading we 
may divide them, as usual, into written examina- 
tions, oral examinations, and practical examina- 
tions ; and, secondly, according to a character 
much less obvious, but more important. They 
may be divided into : — 

(1) Knowledge-tests (or memory-tests),* which 
test the power of re-stating facts and arguments 
of a kind that may be learnt by rote — e.g. “ De- 

•scribe Joule’s experiments on the mechanical 
equivalent of heat ” ; " Relate the chief events 
in the reigri of Edward I.” ; " Write out the 47th 
proposition of the first book of Euclid ” ; " How 
is the preposition in used in Latin ? ” 

(2) What may be called capacity-tests, which 

* C/. p. 5* and NoteF, p..r36, bdow. 

II 



EXAMINATIONS AND 

test the power of dging something, of applying 
knowledge to sonaie definite purpose, e.g. the power 
of making a prScis of a written document, of 
writing a letter or a report on a particular subject 
-with a particular object in view, of translating 
from or into a foreign language, or of conversing 
in that language, of solving a mathematical prob- 
lem, of criticising a passage from a literary work, 
of writing an essay on an historical or literary 
subject with the aid of books in a library, of 
diagnosing the malady of a patient, of analysing 
a chemical mixture or compound ; and (perhaps, 
in certain ways, the highest form under the rubric) 
of making an original contribution to learning or 
science as the result of personal investigatjfta^r 
experiment. 

In some cases the category to which a question 
belongs will depend on the previous preparation of 
the candidate. If a candidate who is asked to de- 
scribe the foreign policy of Quee!^ Elizabeth has 
had access only to text-books in which Elizabeth’s 
foreign policy is treated incidentally in a more or 
less chronological record of events, he may show 
considerable capacity in separating out in his mind 
the threads of foreign policy fron* the rest, and in 
putting the results of his thinking into clear lan- 
guage ; if he has been in the hands of a teacher 
who has dwelt on the particular question, his reply 
may show memory or “ knowledge ” and nothing 
more. In subjects like political history or history 
of literature, it appears to me extraordinarily diffi- 
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cult to set questions in papers of the ordinary 
three-hours type which s)iall coihe clearly under 
the heading of capacity-tests ‘ ; whereas in many 
subjects the questions may be either knowledge- 
tests or capacity-tests, and most written examina- 
tion-papers include questions of both types. 

The classification which I have here set down, 
hijited at by Mark Pattison, and no doubt by 
earlier writers, but perhaps first clearly enunciated 
in a remarkable but too little-known work by 
Latham on " The Action of Examinations,” pub- 
lished in 1877, obviously does not correspond to a 
logical dichotomy. The capacity-test of the kind 
used in examinations does imply knowledge, 
thoHffh the knowledge-test does not imply capacity 
beyond that of memorising, and an elementary (and 
at times doubtful) mastery of the mother-tongue. 
It is sometimes said that the memory-tests set at 
examinations aje too hard ; that the knowledge 
laboriously pumped into the brains of candidates 
under examination-pressure escapes rapidly when 
that pressure has been released ; and that if we 
were to apply the witty definition of education as 
,what remains when we have forgotten what we 
have learnt, we should find the result of examina- 
tion-education to be nil. It would not be wise to 

^ Mr. M. W. Keatinge, in his Studies in History (A. & C, 
Black, 1910, chap, viii.), makes interesting proposals to meet 
these difficulties. Sir Napier Shaw, F.R.S., in an interesting 
pamphlet privately printed, suggests that our curriculum 
is largely determined by the form of the ordinary three-hours* 
paper. 
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accept that conclusion without reserve. It is, at 
any rate, true that examiners do not by any means 
demand that candidates should know everything 
that they are asked ; the ordinary examiner at a 
pass examination is content if the examinee can 
answer satisfactorily from one-third to one-half of 
the questions set. When these questions are 
memory-tests, the indulgence is surely justified <in 
most cases, although there are perhaps ignorances 
of fact of a dangerous character to which this 
leniency should not apply. An examiner once 
told me that on the ground that a certain candi- 
date had obtained the necessary number of marks 
in physiology (and I shall come to the question of 
marks presently), he had been allowed to^pass, 
although he firmly believed that there was air 
in the left ventricle of the human heart. The 


examination, I may say, was not a University ex- 
amination. On the other hand, for some exami- 
ners printing might never have been invented, and 
books of reference might be as rare as they were 
in the Middle Ages. I shall not be accused of 
exaggeration when I say that a knowledge of 
technical terms in foreign languages is sometimes#, 
demanded by examiners which would certainly 
not be expected of candidates in their own 
language. 

In dealing with capacity-tests, the question of 
leniency and severity takes on another aspect. A 
boy who can do right only five addition sums out 


of ten cannot add. A person who reaijjs a ther- 
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mometer accurately five times and inaccurately 
five times cannot read a thermometer. A person 
who understands nine-tenths of the words in an 
easy passage in a foreign language, with or with- 
out the use of a dictionary, but is at sea in regard 
to the meaning as a whole, has not brought 
his knowledge of the language to a useful point. 
On* the other hand, if a candidate can satisfy 
capacity-tests, if we are able to assert from the 
definite evidence of the examination-room, by his 
obtaining not 30 or 50, but, say, 80 or 90 per cent, 
of the maximum marks assigned to such tests, ‘ 
that he can do this kind of problem in mathe- 
matics, that he can speak this foreign language 
so aarto make himself understood on everyday 
topics, that he can analyse a leading article in a 
daily paper so as to present its gist fairly and 
clearly to a person who has not read it, we need 
not fear that th#se capacities will fade away a 
week or a fortnight after the examination. I 
firmly believe that examinations can tell us some- 
thing definite about candidates. The question is, 
do they at the present moment ? It is obvious 
Jihat the answer will not be the same in regard to 
all examinations — that no general answer can be 
given to my question. But we shall understand 
its precise bearing if we deal a little more in detail 

Cf. the evidence given by Prof. Gabriel Lippmann, the 
distinguished Professor of Physics at the Sorbonne, at the 
Enquitt sur V Enseigfttment Secondairt, conducted by the 
Chamber o^eputios in 1899, ProdS’Wrbmx, vol. ii., p, 35, 
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with the question of numerical marks, adopted, 
not in all examinations, but in most pass ex- 
aminations and in most competitive exami- 
nations. 

The system is familiar to every one, A certain 
number of marks is assigned to each question set, 
and the maximum obtainable for a paper is the 
sum of these marks. The total number of marks 
allotted to each candidate is the sum of the marks 
assigned to his several answers to those questions ; 
and the ordinary working hypothesis adopted is 
that the value of the performance of the candidate 
is proportional to the number of marks he thus 
obtains. Putting the thing in another way, we 
may say that the hypothesis amounts t«»this, 
that, if we arrange candidates in the order of their 
performance as determined by the aggregate of 
marks obtained, we shall be doing them no injus- 
tice ; and this order is commoi^y called an order 
of merit (though order of proficiency would per- 
haps be a more accurate expression). 

The value of the hypothesis may be tested in 
two ways. We may examine the postulates on 
which it rests, and the results to which it lead^»^ 
Take, first of all, the postulates. The h5q)Othesis 
implies that we have some reasonable basis for 
determining the relative values of a series of ques- 
tions, i.e. that a number of different examiners, 
regarded as equally competent, will attribute 
approximately the same relative values to the 
individual questions ; and it implies, again, that 
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the same set of examiners will attribute approxi- 
mately the same values to the answers to those 
questions given by a particular candidate. Now, 
although examinations are carried out on such a 
gigantic scale, we have very few published data 
to test the validity of these postulates. In a 
statistical investigation by Professor Edgeworth, 
it appeared that there might be very great dis- 
crepancies in the marks assigned by different 
examiners to the answers of a set of candidates.' 
If this is so, how can the same standard be main- 
tained ? The result is what we might expect in 
the case of examinations of which the object is 
indeterminate, for how can we estimate the value 
of apjjthing unless we know what is its precise 
object ? If a candidate is allowed to pass by 
obtaining from 30 to 50 per cent, of marks on 
memory-tests and capacity-tests mixed in variable 
proportions, what does a “ pass ” mean ? What 
may we say that a man who just passes an ex- 
amination can do ? Is not the meaning of a 
“ pass ” in itself indeterminate ? Does not the 
standard necessarily vary from candidate to candi- 
date as well as from year to year ? 

* If we turn again to the results of examinations 
carried on on a large scale, there are indications 
that the res\ilts of this working hypothesis are 
not more satisfactory than its postulates. Ex- 
aminers themselves are often heard to say, '' This 
man is better than his marks, this man is worse 
See Note E, p. txS, b^ow. 
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than his marks,” * although perfect freedom may 
be left to them to assign marks so as to correspond 
to resultant merit. And if we inspect published 
statistics of examinations for which a large num- 
ber of candidates present themselves, and in which 
it would be reasonable to expect that approxi- 
mately the same number would attain a pass 
standard in consecutive years, they show l^at 
there may be the most extraordinary variations 
in the percentage of passes in the same subject. 
Those variations, as I think any one acquainted 
with the inner working of our examination system 
will agree with me in saying, cannot be regarded 
as due to the carelessness of individual examiners ; 
for, in the first place, examiners take a great deal 
of pains with their work, and, in the second place, 
carelessness may tell as much in favour of a 
particular candidate as against him. I do not 
think that it is at present possible, without further 
scientific investigation on a considerable scale, to 
state what are the precise facts of the case, or the 
precise causes of these variations. They may be 
due to defects in our system of marking ; they 
may be due to real variations in the quality of 
candidates, of which the causes are unknown and^ 
which it would be important to discover. 

I want to strengthen my plea for investigation 
by quoting a witness whom I might have expected 
to find against me. If there is one subject in 
which accurate marking and community of 
‘ C/. p. 6o, below. 
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standard would appear to be most easily obtain- 
able, it is mathematics ; and it seems probable 
that the whole S5rstem of marking by numbers is 
derived from the marking of mathematical papers. 
Nevertheless, Professor Bryan,* a distinguished 
mathematician and a high authority on the sub- 
ject, regards some of the troubles to which I have 
referred as due to " fetish-worship ” of the maxi- 
mum mark, and suggests that we should assign 
certain marks to correct answers of average 
quality, and then mark up to any maximum we 
please, to reward exceptional proficiency shown 
on any part of a paper. In this way, he thinks, 
candidates would no longer feel the inducement to 
" scramble for marks by scribbling scrappy at- 
tempts. ’ ’ Other authorities have suggested differ- 
ent ways of dealing with these difficulties, but it 
is safe to say that there is no agreement, and 
that different examiners employ very different 
methods. The a, 7, S, or Oxford system is 
another form of numerical marking in which the 
total number of marks allotted is low,* and in 
which, if I am rightly informed, the examiners 
^s a rule attempt to obtain their resultant im- 
pression of a paper as a whole directly, instead of 
by the addition of marks assigned to individual 

* Articles on " Examinations,” by Prof. G. H. Bryan, 
Sc.D., F.R.S., published in the School World for May and 
June. 1910. 

* By adding the signs + or — to the Greek letters, the 
number of symbols used is, as a rule, increased to ten or 
more. Cf. p. 62, below. 
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questions. Here, again, it would require investi- 
gation to decide whether this system is or is not 
more trustworthy than the other. In any case, 
in determining a man’s pass or class, you do not 
escape entirely from the difficulties involved in 
the addition of marks. Moreover, the Greek 
letter system appears, so far as I can see, to be 
inapplicable to a competitive examination on a 
large scale in which candidates present themselves 
in a number of subjects.* 

The assignment of marks in practical examina- 
tions is probably an easier matter than in written, 
for the reason that most practical examinations, 
though not all, are capacity-tests rather than 
memory-tests, and it is easier to say whether a 
candidate can or cannot do what is required of 
him. 

Of all kinds of examinations, that which may 
be the most valuable, and that which presents 
the most difficulties, is the viva-i'oce examination. 
Largely used abroad, it is used comparatively little 
in this country, because our boys and girls are not 
taught to speak and to think at the same time 
at the age when children speak confidently and 
easily ; and hence, in the presence of the examiner^ 
the candidate is often paralysed by nervousness, 
for which it is difi&cult, or impossible, to make the 
proper allowance. He may be reduced to silence 
or nonsense. A candidate in Materia Medica in 
a northern University, who was unable, owing to 

I See however p. 62, below. 
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examination fright, to distinguish the taste or the 
smell of the sample of cod -liver oil that he held 
in his hand, on being asked, " Where does cod- 
liver oil come from ? ” replied, " The whale.” 

Nevertheless, the viva-voce examination has the 
immense advantage of allowing a reasonable ex- 
aminer to ask a candidate not paralysed by nerves 
what portion of a subject he has specially studied, 
and to cross-examine him considerately in that. 
The negative evidence that a candidate says 
nothing in regard to a particular printed question 
does not carry one very far. The frank ” I don’t 
know ” of a candidate, his power of distinguishing 
what he does know from what he does not, will, 
with an examiner capable of judging men, count 
very greatly in favour of a candidate whose know- 
ledge of the parts of the subjects that he thinks 
he knows is sound. But a viva-voce examination, 
if it is to be valuable, must not be hurried, and 
hence it is an expensive form of examination. It 
has again the defects of its qualities. The idios5m- 
crasies of examiner and candidate may come into 
conflict in regard to matters which have nothing 
to do with the particular examination-test. To 
combat this defect, it is generally recognised that 
viva-voce examinations should, if practicable, be 
conducted by a board of examiners rather than by 
a single examiner. 

We have in the viva-voce examination a sense of 
reality that is apt to become somewhat thin in the 
written examination. If an examiner knows that 
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,he has to decide, face to face with a candidate, 
not whether he attains a particular standard in a 
particular subject, but whether he has the know- 
ledge and capacity to do a particular job or pursue 
a particular career, he will feel much more able to 
come to a decision than after he has merely seen 
the candidate’s written papers. Would any sen- 
sible person, in business, in war, or in adminis- 
tration, free to choose his own subordinates in 
carrying out an important piece of work, dream 
of doing so without an interview, if he could help 
it ? On the whole, it seems to me that our ex- 
amination system would be more efficient if larger 
use were made of the oral method than is made at 
present. But there are administrative difficulties 
in conducting examinations by that method not 
to be under-estimated. 

Before leaving the question of examination 
methods to inquire further into the effects of 
examinations on the candidates, I want to throw 
out one suggestion. I believe that we should 
efiect a quiet revolution in pass examinations, but 
a revolution, if we would systematically ask our- 
selves in regard to each examination, What does 
it prove that all successful candidates can do ? * 
What are they certainly fit to do ? Such a ques- 
tion would put capacity-tests and memory-tests, 

I believe, in their right places, and the examina- 
tion would become once more, as in its origin, 
what the mathematicians call a directed quantity. 
And now for the effect of examinations on candi- 
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dates. If we take a large cold thermometer, as 
accurate an instrument as we please, and plunge 
it into a relatively small quantity of warm water, 
the reading of the thermometer will not tell us 
directly what the temperature of the water was 
before we tried to measure it. In the same way, 
even the most admirably designed and organised 
examination will generally exercise a very con- 
siderable disturbing influence on the previous 
education of the candidate that it is designed to 
test, and on the candidate himself during the 
examination. In the case of the thermometer, 
we can correct our original reading by making the 
necessary allowance ; in the case of the examina- 
tion candidate, what correction can we possibly 
make ? 

How, if we cannot calculate them precisely, 
shall we reduce the bad influences and magnify 
the good influences of examinations as far as 
possible ? It is" a grave problem, and we shall 
find considerable disagreement in determining 
what features of examinations belong to one cate- 
gory, and what to the other. Let us, for the 
moment, consider the influence of the examina- 
tion on the candidate while it is being actually 
carried on. There can be no doubt that examina- 
tions prolonged over several days, for from four 
to six hours a day or more, involve a considerable 
physical strain on the candidates, and the sample 
of knowledge and capacity which the examina- 
tion affords can hardly be regarded as an average 
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sample. With some candidates the excitement 
and strain make them do better than they other- 
wise would do ; with others the reverse is the case. 
Ought we to try to reduce that strain ? A good 
many, I fear, would not assent. It has been 
said, and not infrequently said, that if a candidate 
fails under the strain, he is lacking in some of 
those qualities which go to make a successful man, 
and that though failure in the particular examina- 
tion does not mean (as the public might, and often 
do, think) that he has not the intellectual attain- 
ments of those who pass, yet that no harm is 
done. It seems probable — one cannot go further 
than that — that failure in examinations owing to 
temporary ill-health takes place more frequently 
in the case of women candidates than men, and 
the question of women candidates certainly needs 
separate investigation ; but in the absence of 
adequate statistical and medical inquiry, the whole 
hygiene of the examination-room Remains obscure. 
Much more important, in the long run, than the 
immediate influence of the examination-room on 
the candidate is the influence of his preparation 
for the examination during the long years of 
school and university life. That ir, a subject too 
complex and important for me to do more than 
mention ; I would only suggest two things — first, 
that in some subjects and with some candidates, 
the influence of examinations may be, and often 
is, good and not bad ; and secondly, that instead 
of encouraging by means of examination subjects 
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ill-suited for examination-tests, we might try to 
protect them from those tests. I do not say that 
it is easy, but I think it not impossible. 

Let me take the subject of history, which, of 
all subjects that come under the heading of 
‘‘ culture ” subjects rather than bread-and-butter 
subjects, is the most important to the ordinary 
citizen. Would it not be possible to reduce 
school examinations in history to a minimum 
by allowing those who had attended a history 
class in a secondary school, to the satisfaction 
of a duly qualified teacher, to be exempted from 
any kind of school-leaving examination in his- 
tory? Of course, it will be said that the stan- 
dards of different teachers would be different, but 
are we not inclined to sacrifice on the altar of 
equality the very efficiency of the teaching that 
examinations are intended to test ? I remem- 
ber, in a French Lyc^e, hearing a lecture on Napo- 
leon’s coup d’etat of the i8th of Brumaire given 
to a class of science boys who were not to be 
examined in the subject. The lecture was full of 
warmth and colour, every boy was interested, 
every boy was listening ; and I contrast that 
with the painful dictation of slow sentences, 
reproducible in the examination-room, that I 
have heard in the history class-rooms of some 
English secondary schools. I do not mean that 
all French teaching of history is good, and that 
all English teaching of history is indifferent. I 
believe thejre is no subject in which more rapid 
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progress is being made in this country. Might 
not the examining bodies help the history teachers 
in other and better ways than by inserting his- 
tory in all their syllabuses, as the teachers seem 
to desire ? 

In dealing with this matter, I would suggest 
that it is as impossible to make certain that every 
boy or girl who leaves school has an adequate 
knowledge of history, has a real beginning of 
historical culture, as it would be to make sure that 
every boy or girl who leaves school has musical 
culture or artistic culture. Supposing you do 
make sure by examinations that the school- 
leaving certificate implies a certain knowledge of 
the facts of history, what then ? A boy coming 
out of the examination-room once said to me : "I 
am never quite sure whether the date of Magna 
Carta was 1215 and Waterloo 1815, or whether 
it was the other way about, but I think I have 
got it right this time." A boy of that kind, even 
with the more difficult questions set nowadays, 
might absorb history, but he would never assimi- 
late it, and instead of tr3dng to force him to be 
cultured in a subject where culture is impossible 
for him, would it not be much better to recognise 
his insensitiveness on the historical side, and push 
him in some other direction ? What I have said 
of history applies even more strongly to literature. 
If we could only make up our minds to require 
real efficiency in one or two subjects, and give our 
boys and girls the opportunity of learning other 
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subjects, without forcing them mechanically to 
do so, would not our education become much more 
effective in the end ? Are we not led by the will- 
o’-the-wisp of general culture into a mire of 
general inefficiency ? I asked a friend of mine 
who is an experienced examiner, what it would 
be safe to assert of the powers of a student who 
co.uld just succeed in obtaining pass marks at a 
pass examination in several subjects. His reply 
was ; “I think you could say that he could pass 
the examination at the time that he passed it." 
I went further. " Would it be safe to say that 
he had any useful knowledge of the subjects of 
the examination — that he would be fit for any 
career ? " The answer was clear : " No, you 
could say none of these things. When you have 
said that he has passed the examination, you 
have said all that can be said." 

If we turn from pass to competitive examina- 
tions, we find that they involve all the same prob- 
lems, and fresh ones to boot. Here it is well 
recognised that difficulty may be used as the 
means, not of testing proficiency in subjects that 
will be required in later life, but to distinguish 
different candidates in regard to what is called 
their " general ability." The great choice of 
subjects permitted by the regulations for the 
Open Competitive Examination for the Home and 
Indian Civil Services sufficiently illustrates this 
point. That particular examination is one of the 
greatest importance. Is it really fulfilling its pur- 
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pose in the best way possible ? Professor E. G. 
Browne, of Cambridge, in giving evidence before 
Lord Reay’s Committee on Oriental Studies 
(Minutes of Evidence, page 62), said that the 
men selected by the method devised by Lord 
Cromer, in connection with the Egyptian Services, 
were more apt in the learning of Oriental 
languages than the men selected by open C051- 
petition for the Indian Civil Service. ‘ I believe 
Lord Cromer intends to say something, later, 
on the methods that he used. I do not say that 
Professor Browne’s isolated observation proves 
anything conclusive, but it is significant, and 
points to inquiry. 

The tremendous severity of the Civil Service 
examinations has been regarded as imposing a 
sufficient test of the character and of the physical 
strength required for arduous public service. But 
do we know at the present moment whether the 
game is worth the candle ? A letter from a 
candidate, who wrote over the somewhat unfor- 
tunate signature “ Presque pass6,” in the Times 
of last summer, reminds one uncomfortably of 
the sad picture of overworked Indian examina- 
tion students drawn about the sanie time by Mr. * 
[now Sir] Valentine Chirol in his articles on Wian 

^ Minutes of evidence of Treasury Committee on the 
Organisation of Oriental Studies in London (Cd. 45^i); 1909, 
Questions 1740-50. Prof. Browne attributed the difference 
to the prominence given to mathematics and science in the 
competitive examination. 
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Unrest.' Is it not possible to test “ general 
ability,” and to separate the ablest candidates by 
methods involving less strain both on the success- 
ful and the unsuccessful candidates ? The ex- 
perimental psychologists and statisticians, work- 
ing on lines suggested by the late Sir Francis 
Galton, hold out hopes in this direction, but one 
mijst confess that they are still somewhat remote. 
Would it be possible, without reintroducing the 
unspeakable evils of jobbery, to follow the lines 
laid down by Lord Cromer in the Egyptian Civil 
Service, and by Lord Selbome in choosing candi- 
dates for the Navy ? These are questions again 
for inquiry.* 

1 Reprinted as a volume, Indian Unrest, 1910, see pp. 217- 
18. 

> Referring to examinations for the Indian Civil Service, 
Army, and Home Civil Service Clerkships of the Second 
Order, Prof. Edgeworth writes (Journal of the Royal Stotts^ 
Heal Society, vol. ift., p. 653) : — ** I find the element of 
chance in these public examinations to be such, that only 
a fraction — from a third to two-thirds — of the successful 
candidates can be regarded as quite safe, above the danger 
of coming out unsuccessful if a difierent set of equally com- 
petent examiners had happened to be appointed.*' With- 
out being an expert in statistics, I venture to think that the 
basis of Prof. Edgeworth's calculations is too slender, and 
now too much out of date, for us to draw general conclu- 
sions from them ; the uncertainty of the examinations in 
question may be to-day much greater or much less than 
that shown by those calculations. The serious matter is 
that we are entirely in the dark as to the actual situation. 
[I reprint this footnote as being an integral part of the 
original paper. For a fuller account of Prof. Edgeworth's 
work, see Note E, p, X02, below.— P. J. H.] 
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Conclusion 

How should such inquiry be carried out ? I 
do not think that any other body than a Royal 
Commission would carry with it the necessary 
weight. I believe the Commission should be a 
small one, presided over by a statesman with great 
experience of affairs, and that there should be no 
attempt to achieve the impossible by including 
in it representatives of all parties concerned. If, 
following the example set in Professor Sadler's 
Moral Education inquiry,* and in the recent report 
of the Consultative Committee of the Board of 
Education on Evening Schools, the evidence 
taken were published, not in the unreadable and 
xmwieldy form of question and answer, but in 
the form of statements on particular topics sum- 
marised by the witnesses from shorthand notes, 
after their examination by the Commission, every 
person and every interest would have a fair say ; 
and the members of the Commission themselves 
would not have any undue advantage in giving 
their own summary of the evidence, and their 
own conclusions. But such a Commission, to do 
all that it should do, would need to be far more 
than a Commission of Inquiry in the ordinary 
sense ; it would need to be a Commission of In- 
vestigation. It should have the power to appoint 

* Moral Instruction Training in Schools: an International 
Inquiry, ed. by M. £. Sadler (Loogmans), 1908. 
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and to pay committees of experts in different 
subjects, who would carry out investigation under 
its direction. Take the questions of standard 
and of marking. Those questions it is difficult to 
investigate adequately without having a large 
number of candidates' answers lithographed or 
printed and sent to different examiners to be 
marked, first without any directions at all, and 
secondly with directions relating to the stress to 
be laid on particular points regarded as of especial 
importance. Nor would the expert in his subject 
alone be necessary ; the assistance of the statis- 
tician would have to be called in at each step, 
I believe that in this way we might be able to 
make an examination standard that would mean 
something very different from what it means now, 
that would be reasonably, though not rigidly, 
constant from year to year, and that would sub- 
stitute the fairqpss of considered judgment for 
the unfair impartiality of chance, in the case of 
far more candidates near the borderline. That 
would be one of the matters for the Commission 
to investigate. If it were to appoint sub-com- 
mittees in a half-dozen, or a dozen, subjects of 
examination, the results of the inquiry would 
probably throw sufficient light on the subjects 
which it was not possible to touch. 

A second question for investigation is the 
hygiene of the examination-room. How far are 
we making fair demands on our students in the 
examinations ? How far do they exceed, from 
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the point of view of strain, what it is reasonable 
to demand ? 

A third point for inquiry, and that the most 
difficult, is the influence of examinations, good 
and bad, on education as a whole. I would only 
suggest here that different classes of candidates 
and different subjects require different treat- 
ment, and that it should be the object of the Com- 
mission to find some way of protecting from 
examination those school subjects which are un- 
suited to them, and in which examinations dig 
up the roots of the plants which we wish to grow. 

Fourthly, the Commission would inquire how 
far our different sets of competitive examinations 
select the persons most fitted for the posts which 
they are intended to occupy. 

In this connection, the Commission would, no 
doubt, consider the question of associating as 
examiners with the experts in^ their particular 
subjects administrators with experience of the 
services for which candidates were being chosen. 

I have selected four large topics for inquiry : 
(i) the question of standards and marking, with 
which is closely connected the question of the 
precise object of the examination concerned ; (2) 
the hygiene and psychology of examinations ; (3) 
the influences of examinations, good and bad, on 
previous training ; (4) competitive examinations 
for the Services. But these topics would sub- 
divide, and others would suggest themselves for 
investigation. A Commission with a suitable 

3a 



NATIONAL EFFICIENCY 


reference would find no lack of useful work to do. 
I suggest as a reference ; " To investigate and 
report upon the methods and efficiency for their 
purpose of examinations carried on by Govern- 
ment Departments a id other public bodies in the 
United Kingdom ; to inquire into the influences 
of examinations on the previous education of 
candidates ; and to suggest such changes as may 
seem desirable.” 

The investigations of such a Commission would 
be of immense assistance both to the teaching and 
to the examining bodies of this country. I have 
tried to obtain figures showing the magnitude of 
our examination operations. Those figures are 
still far from complete, but I have official statistics, 
supplied to me by the authorities concerned, which 
show that in the year ending September 30, 1910, 
out of 146,741 candidates at public examinations 
in the United Kingdom, 71,114 failed. I have 
little doubt tha? when my figures are complete — 
and I hope I may be permitted to publish them 
later — the total entries will approach 200,000 and 
the failures 100,000.* Does there not seem to be 
something very wrong in the relations of our 
examinations with our education for nearly half 
the persons to be judged imfit to pass the test for 
which they present themselves ? How far is this 
great percentage of failures due to a low average 

i Several bodies which were good enough to furnish me 
with statistics made it a condition that the details should 
ncit be published, 

D 
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of intelligence or industry, how far to the fault 
of the teachers, or the fault of the examiners, or 
to want of co-ordination between teacher and 
examiner? Conscientiously conducted in their 
details, have not our examinations been largely 
at the mercy of hazard in regard to principles ? 

Lord Rosebery, speaking in 1907 as Chancellor 
of the University of London,* urged strongly that 
examiners needed training no less than teachers. 
He also suggested the need for investigation. 
For such teaching as Lord Rosebery desires I 
doubt if any common basis of doctrine at present 
exists. It must be created. 

There is one serious obstacle to change. The 
examiner and the organiser of examinations are 
often so weary of their ungrateful tasks that for 
them the day’s work is enough ; they become 
hardened, sceptical of improvement, reconciled to 
evils that they have come to regard as unavoid- 
able evils. But the issues are too grave to be 
overlooked, and the investigation and reform that 
I have ventured to demand are a matter of 
national necessity. 

Speech by the Late Earl of Cromer, 
O.M., G.C.B. 

Lord Cromer, speaking from the Chair, said ; — 
I am sure that you will all agree with me when I 
say that we owe a deep debt of gratitude to Mr. 

* At the openioig of the London Day Training College : tee 
the Timts of November 4tb, 1907. 
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Hartog for the very interesting paper to which 
we have just listened. It is a paper which stimu- 
lates thought, and, let me add, there are few sub- 
jects more deserving of thought at present than 
that to which Mr. Hartog has drawn our atten- 
tion. We seem to be about to enter a period 
during which, whether for good or evil, the State, 
either in its imperial or municipal capacity, is 
likely to develop great activity in the perfor- 
mance of a variety of new functions. How are 
the large additional number of agents whose 
services will thus be required to be recruited ? 
The results which will be obtained by the new 
departure now being made will, in a great measure, 
depend upon the answer which is given to this 
question. It is perhaps an exaggeration to say 
with Pope that " Whate’er is best administered 
is best,” but it is certainly true that a good 
system may be jnarred by mal-administration, 
whilst the defects of a faulty system may be 
greatly mitigated if the administration is efficient. 

There are, so far as I know, only three methods 
under which public functionaries can be ap- 
pointed. These are election, selection or nomina- 
tion — the two words are, for the purposes of my 
present argument, synonymous — and competition, 
either open or restricted. 

I need say but little of the elective method. Its 
advantages are mainly political rather than 
administrative. An eminent French economist, 
M. Leroy Beaulieu, has said : ” According to 
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democratic theory, popular election will always 
secure the most capable individuals ; but this is 
a curiously mistaken idea, and is quite unsup- 
ported by experience.” 

The Poor Law Commission, over which Lord 
George Hamilton recently presided, took the same 
view. ” The method of popular election,” the 
Commissioners reported, ” is not one which is 
adapted to get the best administration. The 
Commission believe that there is a better chance 
of good administration if the members are 
appointed than if it is left to the chance of elec- 
tion.” I will not, however, dwell any further 
on this branch of the subject, as it touches the 
fringe of political issues, which I am anxious to 
avoid. I will only, therefore, say in general 
terms that one, and perhaps not the least impor- 
tant of the questions which the democracy of 
this country will have to face i» the near future, 
is how to combine administrative efficiency with 
an adequate amount of popular control. I have 
often, in the course of my Eastern experience, had 
to deal with a very similar problem, and I gener- 
ally found that the question practically resolved 
itself into this — How much efficiency is it 
justifiable to sacrifice on the altar of self-govern- 
ment ? I came to the conclusion, under the 
special circumstances with which I had to deal, 
that it was impossible to generalise on this sub- 
ject, and that each separate issue, as it arose, had 
to be decided on its own merits. 
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Turning to the other alternatives, viz., selection 
br competition, it may readily be admitted that 
competition, which is the prevailing system now, 
has its defects, but before condemning it, it is 
worth while to glance back at the system which 
it supplanted. I am one of the band, not now 
very numerous, who can speak with some personal 
experience of the pre-competition days, and, as 
illustrations of the working of the then existing 
system, I may perhaps be pardoned if I relate a 
few biographical details. Some fifty-six years 
ago, being then a boy of fourteen, I was admitted 
into the Royal Military Academy at Woolwich, 
with a view to eventually becoming an officer in 
the Artillery or Engineers. At that time there 
was a sort of sham entrance-examination, but I 
never heard of any boy who had been nominated 
by the Master-General of the Ordnance being 
rejected. Lord Hardinge, the grandfather of 
the present Vice*Ioy of India, who was a friend of 
my family, gave me my nomination. On pre- 
senting myself at Woolwich for medical examina- 
tion, I was very rightly rejected for short sight. 
I returned to London and told my mother, 
who was my only surviving parent. She acted 
with promptitude. She instantly rang the bell, 
ordered her carriage, and went to the Horse 
Guards to see Lord Raglan, who had succeeded 
to Lord Hardinge's place, and who was another 
friend of my family. The result was that next 
day I returned to Woolwich with a letter addressed 
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by Lord Raglan to the medical officer, asking him 
to " reconsider the matter.” I was, of course, 
admitted . Exactly the same thing happened at the 
same time to another lad who was at first rejected 
on the ground that he had a serious impediment 
in his speech, but whose case was subsequently, 
under pressure, ” reconsidered.” This sort of 
thing could not happen now. The practical 
working of the system may, however, best be 
illustrated by an anecdote which is related of 
Lord Melbourne. A friend of his who occupied 
a high position and who disposed of a good deal 
of patronage, said to him : ” I do not in the least 
mind confessing that if I had to deal with two 
candidates, one of whom was the son of a friend 
or relation of mine, and the other a stranger, I 
should, ceteris paribus, give the appointment to 
the son of my friend or relation.” To which 
Lord Melbourne drily replied, ” So should I, but 
ceteris paribus be d— - — d.” In iact, at that time, 
the principle laid down by George III had only 
been but slightly modified. That monarch is 
believed to have said that any one was fit to occupy 
any place he could manage to get. 

Well, ladies and gentlemen, as a result, partly 
of the attention called to military matters by 
reason of the administrative breakdown during 
the Crimean War, and partly of the extraordinary 
call for officers occasioned by the Indian Mutiny, 
the nomination system, in so far as appointments 
to the Artillery and Engineers was concerned, 
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was swept away, and the competitive system intro- 
duced. It started under rather bad auspices. 
I remember that the first notice issued by the War 
Office was rather unfortunately worded. They 
called for " persons ” to compete. The result 
was that all the young men who responded to 
this call were instantly dubbed " persons," and 
commanding officers showed a marked preference 
for those who had entered under the old system, 
and who had enjoyed the more exalted title of 
" gentleman cadets.” I was at that time an 
ardent young Liberal, as I claim to be an old 
Liberal now — although my right to that desig- 
nation may perhaps not be recognised by mere 
party politicians — and I was warmly in favour of 
reform. As a matter of fact, the gloomy pre- 
dictions of the anti-reformers were falsified by 
the event. The social class from which the officers 
who entered th^ Army were drawn was just the 
same after as it had been before the change of 
systems, whilst there can be no doubt that a dis- 
tinct improvement was effected, both in intel- 
lectual standard and professional capacity. 

I mention these points mainly in order to show 
that the competitive system, whatever may be its 
defects, is greatly superior to that which it super- 
seded, and in order to warn the advocates of 
reform that they must beware lest a door be 
opened which may lead to a revival, perhaps in 
a somewhat different form, of the abuses of the 
past. 
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X And now let me make a few remarks on the 
competitive system considered on its own merits. 
I think the best thing I can do is to deal with facts 
which have come within my personal experience ; 
and, as you are probably aware, my personal 
experience has been gained more in the outlying 
portions of the Empire than at home. I have 
seen a good deal of the work performed by tjie 
young men who are turned out by our universities 
and military colleges. I have met them on the 
banks of the Ganges, the Indus, the Irrawaddy, 
and the Nile. And I have also had some rather 
exceptional opportunities of comparing their 
aptitudes with those displayed by their Conti- 
nental rivals and competitors. From one point 
of view the comparison is disappointing. I have 
often had it said to me by heads of departments 
in Egypt, themselves Englishmen, that the young 
Frenchman or German came to^them crammed 
to the muzzle with an amount of preliminary 
knowledge greater than that possessed by the 
young Englishman. The explanation is, I think, 
that until recent years sufficient attention has not 
been paid in this country to technical education ; 
that is unquestionably a defect, more especially 
at a time when there is a tendency to specialise 
every branch of learning, but it is a defect which 
I hope and beUeve is in process of being remedied. 
But if we look at education not merely as a means 
for storing the mind with knowledge, but also as 
an instrument for developing the character, the 

40 



NATIONAL EFFICIENCY 

situations are reversed. The youth in whom the 
need of individual effort and the sense of personal 
responsibility have been nurtured in one of our 
public schools or colleges, becomes almost uncon- 
sciously capax imperii. He is no automaton. 
In the free atmosphere in which his boyhood and 
youth are passed, he learns a number of lessons 
which stand him in good stead in after life as a 
member of an imperial race. More especially, he 
learns not only to think, but, which is much more 
important, to act for himself — in a word, to 
govern, and to govern generally with firmness, 
justice, and wisdom. From this point of view 
our system of making appointments, which is 
generally, though not exclusively, the result of 
competition, may, I think, in spite of some defects, 
be said generally to have produced satisfactory 
results. A very high authority on Indian ahairs, 
Mr. Valentine C^hirol, recently wrote : “ To the 
Indian civilian belongs the credit of almost every 
measure passed during the last fifty years for the 
benefit of the Indian masses, and passed fre- 
quently in the teeth of vehement opposition from 
the Indian politician.” I think you will agree 
with me that this is vety high praise. The system 
has thiK turned out a number of young men who, 
as the agents of a nation whose very existence 
depends upon the execution of a sane imperial 
policy, are not only of incomparable merit, but, 
I do not hesitate to add, are also the envy and 
admiration of the whole civilised world. What- 
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ever changes may in the future be made in our 
educational system, I most earnestly trust that 
no steps will be taken which may tend, in 
any degree, to weaken that sense of independ- 
ence in thought and action which should con- 
stitute the basis of character to the individuals 
who are the units of any self-governing com- 
munity. . 

In spite, however, of this success, I think we 
may well ask ourselves, after the experience of 
half a century of the working of the competitive 
system, whether it is not capable of improvement. 
The question is certainly worthy of consideration. 
Remember that the principle of selection, if only 
it can be properly carried out, possesses merits 
superior to those of competition. The former 
may, or ought, to result in the creation of leaders 
of men. It should enable us to acquire for the 
public service the aristocracy, iq the proper and 
original sense of the term ; that is to say, not the 
aristocracy of birth, but that of intellect and of 
character. The latter tends rather to produce a 
dull level of mediocrity. Most educated English- 
men have recently been reading the history, 
written by a very distinguished contemporary 
statesman, of one of his most illustrious predeces- 
sors — I mean Lord Rosebery's history of the 
early life of Chatham. Now, what was it that 
enabled Lord Chatham to revive British spirit 
when it had sunk to a very low ebb, to march to 
victory on the banks of the St. Lawrence and the 
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Ganges, and to found the British Empire ? I 
say that it was the adoption of the principle of 
selection, sparingly, indeed, but very skilfully em- 
ployed. I notice with pleasure that of late years 
there has been a tendency, notably in the military, 
naval, and diplomatic services, to adopt this 
principle in dealing with all the later stages of the 
cafeers of public servants more thoroughly than 
was formerly the case. I trust that this move- 
ment, far from being arrested, will be pushed 
still further. I could name numberless instances, 
especially in the military service, in which the 
dead weight of rigid regulations, based generally 
on the claims of seniority, have acted detrimen- 
tally to the public interests. 

The case of first appointments presents natur- 
ally far greater difficulties, for, ex hypothesi, there 
can in these cases be no record of past services 
on which to ^ely, and moreover, here, and 
indeed in any application of the principle of selec- 
tion, we are at once met with the awkward 
question, Quis custodiet ipsos cusiodes ? Who is 
to select the selectors ? 

In connection with this subject, there is one 
point in respect of which I cannot help thinking 
that a good deal of misapprehension exists. It is 
not unfrequently thought that Ministers and 
others in whose hands the bestowal of patronage 
is vested, would rather welcome an accession to 
their powers in the matter of making first appoint- 
ments. I believe this view to be wholly erroneous. 
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My experience has led me to the conclusion that 

those who exercise patronage are also those who 

most of all delight in being fettered by regulations 

which relieve them of a very irksome and onerous 

duty, in the exercise of which they are certain 

to incur a great deal of misrepresentation and 

abuse, and to receive very little gratitude. It is, 

in fact, true that the wider is the discretion left 

» 

to a Minister the more fully will he be exposed 
to the full blast of social, and also — which is 
perhaps even more hurtful — of parliamentary 
pressure, to make his selections on grounds 
other than those of merit. Moreover, unless he 
be an exceptionally strong man, it is well-nigh 
certain that he will at times succumb to this 
pressure. More than this, it has to be remem- 
bered that neither the action of Parliament itself, 
in its collective capacity, nor the power exercised 
by the Press, however honest in intention, affords 
any adequate guarantees either against deliberate 
jobbing, or what is perhaps a greater danger, 
against unintentional negligence. The truth is 
that, save in a very few exceptional cases, the 
number of persons who can speak with impar- 
tiality and full knowledge of the merits of a candi- 
date, whether for promotion, and still more for 
first appointment, is very limited, and, njoreover, 
it is not unfrequently difficult to obtain a frank 
expression of their opinions. I do not suppose, 
therefore, that any one, however fully he may 
recognise the defects of the competitive system, 
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would be prepared to advocate a return to the 
principle of uncontrolled selection. 

Let me again fall back on my personal experi- 
ence, and explain how, in one special instance, to 
which Mr. Hartog has alluded, I solved the diffi- 
culty myself. Some few years ago it became 
necessary to create a Soudanese Civil Service. 
In* the first instance the appointments were 
practically made by myself. I found it a very 
difficult task. I used to receive hundreds of 
letters from the friends or relations of candidates. 
Of these about go per cent, found their way into 
the waste-paper basket. I took an enormous 
deal of trouble to select the best men. Indeed, I 
may remark incidentally that whatever success 
has attended the administration of Egypt during 
the last thirty years has been mainly due to the 
great care which was taken both in selecting and 
in promoting officials. Generally speaking, the 
method adopted in the first instance in the 
Soudan for making first appointments worked 
fairly well. Nevertheless, I felt that, apart from 
the great trouble and responsibility it imposed on 
myself, the system, or perhaps I should rather 
say, the absence of system, was very unsatis- 
factory. In the first place, I never could feel 
sure that the merits of many young men perhaps 
more deserving than those who were actually 
appointed had not been overlooked. In the 
second place, although I felt quite certain that I 
should not job, I felt almost equally certain that, 
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sooner or later, I should be accused of jobbing 
and the accusation would in itself have done 
harm, inasmuch as it would have shaken public 
confidence. On the other hand, I was not pre- 
pared to resort to open competition, as I was 
convinced that many of the qualities most re- 
quired for a Soudanese civil servant could no 
be tested by a competitive examination. Even- 
tually a system was adopted under which the dut> 
of selecting candidates, who are all taken frorr 
the universities, was, under conditions whicl 
I need not describe in detail, conferred on a boarc 
of highly qualified officials, who were left quite 
free to act as they thought best in the public 
interests. The system produced good results 
but I do not at all say that on that account it ie 
suitable for general adoption in this country 
There are alternatives on which I need not dwell 
My present object is merely to* draw attentior 
to the subject, and to indicate the advisability 
of considering whether it is not possible to ^.dopt 
measures which will on the one hand admit o 
some independent control being exercised over 
State authorities, and more especially loca 
authorities, in cases where the competitive 
system, properly so called, is not applied, anc 
which will, on the other hand, tend to. remedy 
or at all events to mitigate, the acknowledge^ 
defects in the competitive system itself. The 
test supplied by that system, though in some 
respects valuaWe, is purely intellectual, and 
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indeed, as at present applied, it may be doubted, 
more especially after th^^vidence adduced by 
Mr. Hartog, whether, even -as an intellectual 
test, it always succeeds in attaining its object. 
Time does not permit me to dwell'on the details ; 
I will therefore only say that, in laying down the 
conditions under which a competitive examina- 
tion. is to be conducted, the most important 
points to be considered are, first, how are the 
examiners to be appointed, and, secondly, on 
what principles should they proceed in awarding 
marks. 

For the rest, I should like to point out that 
the Order in Council of 1870 contemplates the 
possibility of a departure from strict rules in 
certain cases, but such a departure can only be 
made with the consent of the Civil Service Com- 
missioners. It is in the development of this idea 
that, for my owi^ part, I should be inclined to 
look for reform, hi any case, I hold a strong 
opinion that if anything is to be done in this 
direction. Parliamentary politicians should be 
rigorously excluded from participation in - any 
Boards or Commissions which may have the 
isposal of patronage. 

Ladies and gentlemen, I fear I have detained 
you too long. I trust, however, that this discus- 
sion will lead to the matter on which Mr. Hartog 
has dwelt being considered by others more com- 
petent than myself to deal with them. 

The idea of appointing a Royal Commission to 
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examine into the whole subject is certainJy one 
which commends itself to me. 

[Note . — The paper and speech printed above, together 
with the discussion which followed, were published in the 
Journal of the Royal Society of Arts for February 3rd and 
loth, 1911. In the course of the discussion the proposals 
for an investigation of the examination-system were sup- 
ported by Sir Henry Miers, F.R.S., Professor John Adams, 
and Professor Arthur Schuster, F.R.S. ; they were, also 
warmly supported by Professor M. E. Sadler in a letter 
printed in the Journal for February 3rd. I have made a 
few slight corrections in the text of the paper, and added one 
or two footnotes.] 



II 


. THE THEORY OF EXAMINATIONS 
I. Definition 

1 . For the purposes of this article the word 
“ examination ” (except where otherwise indicated) 
is used to mean " a systematic test of knowledge, 
or of either special or general capacity or fitness, 
carried out under the authority of some public 
body.” The argument, for the sake of brevity, 
mainly refers to examinations as carried out in 
the United Kingdom. 

11. Origin and Objects of Examinations 

S. Our modem examinations are based on 
mediaeval examinations designed with a view to 
testing whether a candidate had the necessary 
qualifications to practise a craft or profession. 
They have since been applied to the following 
new purposes : (i) to arrange candidates in order 
of merit (competitive examinations) ; (2) to test 
proficiency in a subject, or in one or more sub- 
jects, and, following on this, to test "general 
culture ” ; (3) to test the efficiency of schools. 
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3. The distinction between professional ex- 
aminations on the one hand, in which the exa- 
miners are guided by a definite and definable 
(though often complex and unanalysed) object in 
view, and non-professional examinations on the 
other, in which there is no such object in view, 
serving as a guide, is of a fundamental character. 
Most of the anomalies of our examinations are 
found in non-professional examinations, or in 
those parts of professional examinations based on 
the model of non-professional examinations, 

4. Intermediate between professional and non- 
professional examinations are examinations of 
which the purpose is definable, but of which the 
definition of purpose conveys widely differing 
ideas to different minds. The chief examina- 
tions in this category are (i) " School-leaving 
examinations,” (ii) professional entrance examina- 
tions, and (iii) University entrance examinations 
(Matriculation Examinations). The first are 
supposed to show evidence of a good general 
education, and to imply fitness to enter a 
business or professional career, without specific 
limitations ; the second, to imply fitness to enter 
on a course of study for a particular professional 
career {e.g. that of doctor or solicitor) ; the third 
to imply fitness to pursue a University career. 
Frequently one and the same examination is 
used for all three purposes. 

8. The ejcamination system has acquired an 
immense importance in all civilised States, and 
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1ms become an essential factor in the social 
organism, since the selection of individuals to 
fulfil various functions in the State, as well as 
for the great categories of social occupation, is 
largely determined by examinations. It has been 
calculated {Journal of Royal Society of Arts, 1911, 
p. 1132) that there were in the United Kingdom 
in 1910 over 300,000 examination candidates in 
elementary and secondary subjects only, apart 
from University and professional examinations, 
and the relative numbers of candidates in the 
United States, France, Germany and other large 
countries are probably not less. 

6. Nevertheless, there are few questions relat- 
ing to examinations which can be regarded as 
uncontroversial, owing probably to two reasons : 
(i) the uncertainty of social and educational 
ideals at this time of unrest (even before the 
War); and {2) the fact that examination tests 
have never been submitted to any scientific 
investigation, and that we are largely in the dark 
as to their validity for the purpose for which they 
are employed, and sometimes even as to the pur- 
pose itself of a given examination and the mean- 
ing of the certificates! conferred in connection 
therewith. 

III. Classification of Examinations 

7 . The kinds of test used in examinations may 
be classified for different purposes in a number 
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of ways (which necessarily overlap to some 
extent), as follows : 

(a) According to the method by which answers 
to the examination-questions are furnished by 
the candidates: (i) written, (ii) viva-voce, (iii) 
practical (including clinical). Examinations may 
combine these tests in various ways. 

(b) According to the period occupied by .the 
examination work, which may (i) be relatively 
short and limited by the examining body, extend- 
ing over a period of from a few hours to a few 
days, or (ii) extend over a long period, generally 
without any maximum limit. Examinations 
of course-work done under the supervision of 
teachers, and of theses and dissertations, come 
into this second category. 

(c) According as they are conducted (i) by per- 
sons not appointed to examine as teachers, i.e. ex- 
ternal examiners — it is to be nojed that in some, 
though not in all, external examinations the 
examiners may and do include persons who have 
taught some but not all of the candidates — or 
(ii) by teachers of the candidates, or (iii) by a 
combination of categories (i) and (ii), a frequent 
combination in University examinations in the 
United Kingdom. 

(d) According as they are (i) knowledge (or 
memory) tests, or (ii) capacity-tests, or (iii) 
a combination of the two. A knowledge-test 
shows the power to reproduce in a morfe or less 
modified form information acquired by the can- 
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didate ; a capacity- test shows the power to do 
^mething other than to reproduce, e.g. to speak 
a foreign language, to read a thermometer, to 
auscultate a patient. 

(e) There are also psychological tests, con- 
ducted by experimental methods. 

(/) Tests of medical fitness. 

IV. Criteria of Tests 

8. The word examen means the pointer of a 
balance ; the analogy of the testing of the 
physical properties of a material substance with 
the testing of the fitness and capacity of a human 
being is obvious, and the criteria of physical tests 
can and ought, with certain important limita- 
tions and extensions, to be applied to examina- 
tions of human beings. The principal criteria 
for physical testT' may be stated as follows : 

{a) The test used of a given characteristic 
should demonstrably yield a suitable measure of 
that characteristic. This is not a self-evident 
propositwn (thus it is practically impossible to 
measure directly such, a property as “ absolute 
temperature,” as defined in thermo-dynamics), 
and it is necessary to show how far indirect 
measurements are valid for their purpose (see 
under (ii), § 9, below). Many examination 
methods are indirect tests of the characteristics 
which it is desired to measure and compare. 

(li) In cases where the Whole of the material 
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under investigation cannot be tested, it must be 
shown that the sample tested is an average 
sample of the whole. 

(c) The test must be " trustworthy,” i.e. it 
must, when repeated a number of times as nearly 
as possible under identical conditions, by the 
same observer, or when repeated by different 
observers, yield approximately the same reSults 
(the competence of the observers being assumed). 

{£) The application of the test itself must not 
sensibly affect the characteristic tested, or in- 
juriously affect the material as a whole. 

9. For the sake of brevity we shall, in what 
follows, use the word " characteristic,” as applied 
to examinations, in a wider sense than usual, to 
denote anything for which marks may be given 
to a candidate, e.g. knowledge of French, or of 
mathematics, capacity to ride a horse, etc. The 
numerical evaluation of human characteristics 
presents at least two kinds of difficulty which 
exist only to a much less extent with physical 
characteristics. 

(i) Whereas we can in many (though not all) 
cases legitimately represent the values of physical 
characteristics either in terms of a unit, by 
straight lines of different lengths, or by numerical 
ratios, such representation in the case of human 
characteristics, though convenient, is often of 
doubtful validity. There is no such thing as 
“unit knowledge of mathematics,” or “unit 
knowledge of French,” nor could we justify a 
c 54 



THEORY OF EXAMINATIONS 


. statement that “ candidate A knows twice as 
much French or mathematics as candidate B.” 
We shall deal further with this matter below. 

(ii) The variation of human characteristics both 
with lapse of time and with the conditions under 
which human beings are normally placed is much 
greater, and less easily determinable than the 
variation of the physical characteristics of 
material substances in daily use concurrently 
with the variations of the physical conditions 
under which they are usually placed. Without 
going into an elaborate analysis, we may say that, 
while it is useful for practical purposes to say 
that the density of a piece of iron is 7 8 , by 
which we mean that it differs little from that 
number under every-day conditions of tempera- 
ture and pressure, it is more difficult to make a 
corresponding rough-and-ready statement in 
regard to the important mental characteristics, 
either general, such as " power of attention,” or 
specific, such as ” power to give an account of 
the Norman Conquest,” since these characteristics 
vary under every-day conditions of interest and 
fatigue, and the law of such variations escapes 
definition. 

10. It is clear that the knowledge of a charac- 
teristic of a purely evanescent kind is of little 
value. A candidate who knows the dates of the 
accession and death of the English sovereigns 
merely as a series of munbers on one day may 
have forgotten them on the next. Unless, there- 
in 
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' fore, there is collateral evidence of the likelihood 
of the permanence of the knowledge itself, the 
only permanent or semi-permanent characteristic 
o.f which such knowledge shows evidence is the 
power to memorise numbers, and if we are ignorant 
as to the time taken to memorise them the char- 
acteristic is one to which we are not entitled to 
attach any great value on the evidence shown ^ 
We may add to the criteria of examination tests 
enumerated above the following : {e) the char- 
acteristic tested should be demonstrably of a 
permanent or semi-permanent nature. 

It frequently happens that, while examiners 
nominally assign marks for knowledge, they are 
perfectly aware that the knowledge may be 
evanescent, and they really intend to assign 
marks for what is assumed to be the permanent 
power of acquiring knowledge. 

We shall use some or all of the above criteria 
in dealing with a few of the more important prob- 
lems which arise in connection with examinations. 

V. The Problem of Purpose 

11. Before dealing with more special problems, 
we have to deal with the general problem of pur- 
pose, the most important, but, in very many 
cases, the most neglected of all examination 
problems. The majority of discussions about 
particular examinations, other than final profes- 
sional examinations, are confused because the 
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object and purpose of these examinations is not 
defined, and in the case of “ pass ” and “ honours ” 
examinations because the examiners do not tell 
us what a person who just “ passes ” or just gets 
“ third ” or “ fourth-class honours,” knows or can 
do. From experience we know that we cannot 
say with any certainty that a person who 
” p^LSses ” in a given subject will even be able 
to pass in that subject a year later. Probably if 
the examiners were pressed they would admit 
that ” passing ” the examination means nothing 
more than that a certain candidate was examined 
by certain examiners and was able to earn a 
certain percentage of marks in a certain number 
of subjects at a certain date. We have therefore 
next to investigate the problem of marks and 
marking. 

VI. The Problems of Marking 
© 

12. Most examinations are made up of a series 
of tests, either of the same or different kinds. A 
number of marks is assigned to each question in 
each paper, or viva-voce, or practical examination 
set, and the maximum obtainable is the sum of 
the marks so assigned.’ The total number of 
marks allotted to each candidate is the sum of 
the marks assigned to his several answers to the 
questions which he is asked to answer. The 

^ If a candidate is required to answer only a certain irac* 
tioa of the whole number oi questions, the maximum is 
reduced accordingly* 
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ordinary working hypothesis adopted is that the 
value of the performance of the candidate is 
proportional to the number of marks he obtains, 
and that we shall be doing no injustice to the 
candidate as a whole, if we arrange them in an 
" order of merit ” (better expressed as order of 
proficiency) in accordance with the numerical 
sums of such marks. In pass examinatioiis a 
candidate is ordinarily required to obtain from 
30 to 50 per cent, of the grand maximum assigned 
to all the tests which he is required to take, in 
order to be allowed to “ pass.” 

To satisfy criterion (c) § 8 above, we should be 
able to show (i) that at a given examination a 
number of examiners, regarded as equally com- 
petent, will assign approximately the same relative 
mark-values to the different questions: (ii) that 
the same, or another set of equally competent, 
examiners will attribute the sa^ie marks to the 
several answers to those questions given by any 
Candida te.‘ Further, if we are to be able to 
compare the proficiency of candidates of any year 
with the proficiency of those of a preceding year, 
we must have some adequate coniparison of the 
” standard ” at the two examinations in question, 
an exceedingly difficult and complex problem, of 
which nothing but a rough-and-ready solution 

^ And that the same examiner will allot the sajme marks 
ii he marks the papers a second time. In examinations with 
hundreds or even thousands of scripts to mark> the question 
of the examiner's fatigue becomes of importance. (See 
Note E, p. III, below.) 
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can be provided. It has been proposed in 
examinations at which large numbers of candi- 
dates present themselves to ensure a constant 
standard of “ passing ” by allowing the same per- 
centage to pass each year, thus assuming the 
variations of marks obtained to be due to varia- 
tions in the standard of the difficulty of the 
questions and of the marking of the examiners, 
or both. 

18 . The published evidence in regard to con- 
stancy of marking, notwithstanding its funda- 
mental importance, is exceedingly slender. Edge- 
worth in 1888-90, in an investigation on First 
Class Civil Service Examinatiohs, found the 
element of chance in the marking to be so great 
that only a fraction of from a third to two-thirds 
of the successful candidates could be regarded 
as above the danger of coming out unsuccessful 
if a different mt of equally competent examiners 
had been appointed.* The element of chance 
may be expected a priori to vary with the purpose 
of the examination and to be least in a “ final ” 
professional examination, where, in awarding 
marks, the examineri has more or less consciously 
in his mind the relative importance of the tests 
in relation to the requirements of the profession 
and the fitness of each candidate to enter the 
profession. Where, as in a pass examination in 
“ arts ” or " science,” the purpose of the exa- 
mination is not defined, the meaning of a “ pass ” 

1 See Note E, p. 129, below. 
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is necessarily indeterminate. Corresponding to 
this indetermination we find remarkable varia- 
tions in the percentage of candidates who pass 
from year to year. It may be safely assumed that 
these variations are not due either to variation 
in the percentage of marks required for a pass, 
or to carelessness on the part of the examiners. 
Any variation in the percentage of marks required 
for a pass is probably made with a view to reducing 
rather than to increasing differences in the per- 
centages of “ passes ” from year to year ; and 
carelessness may easily tell in favour of as well 
as against any particular candidate.^ Without 
careful statistical investigation on a large scale 
it is impossible to state what are the precise facts 
of the case or the precise causes of the variations 
of passes. They may be complex, and due (a) to 
real variations in the average proficiency of the 
candidates, or {b) to variations ip the methods 
of setting papers and of marking the answers by 
the examiners, or (c) to a combination of these 
causes. 

14. We have assumed, so far, that the total 
marks earned by a candidate are the sum of the 
marks earned on the different parts of the examina- 
tion, but it is shown by experience that exami- 
ners are frequently not satisfied that the totals of 
marks assigned in the way described represent 

> In well organised examinations special precautions are 
taken to eliminate error through marks being wrongly added 
or overlooked. 
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the proficiency of thi candidates, and are apt 
to say “ the candidate is a good mark-getter, or 
a bad mark-getter as the case may be.” ‘ Pro- 
fessor Bryan, F.R.S., as a remedy,* has pro- 
posed to abolish absolute maxima, and to leave 
examiners free to add on marks for exception- 
ally good answers. The same result might be 
mere conveniently achieved by assigning (say 
20 per cent.) marks for general impression, so 
that the total marks might be made to corre- 
spond to the ” resultant impression ” of the 
examiner, which is probably, though unanalysed, 
a safer guide than the mere addition of marks, 
in the existing absence of statistical information 
as to the general trustworthiness of the marking 
system. Some examiners give negative marks 
for answers showing want of comprehension, but 
the practice is not general. Experience seems to 
show that examiners are more frequently in agree- 
ment as to the “ class ” of a candidate than as to 
his marks, the number of ” classes ” assigned being 
not more than four. In the minds of examiners 
there sometimes appears to be a real if undefin- 
able distinction between the classes of two candi- 
dates, A and B, who have obtained the same 
numerical marks in the first instance ; they may 
say A is a ” first class ” man, B is a ” second 
class " man ; and the separation of the two may 
be finally effected by an adjustment of marks. 

^ Stanley Leathes, Whut is Education ? p. 154. 

* Loc, cit. See note, p. 19. 
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Possibly it might be sKown that examiners 
in this way distinguish between evidence of 
" capacity ” and of " memory ” or knowledge. 

16. In the " Greek-letter ” system of marking 
each answer is marked a, 7 , or S, in the first 
instance, with (say) four further distinctions 

(+. + +, — , 1 ) attached to each of the first 

three letters. The system has some advantages, 
but it is difficult to see how it can be applied to 
a competitive examination on a large scale unless 
the symbols are translated into numbers.’ It has 
been suggested that the only satisfactory way of 
determining " order of proficiency ” (or " merit ”) 
is ta compare each candidate’s performance with 
that of each of the others and to assign a final 
order in accordance with the number of superi- 
orities shown by the several candidates. The 
process is too laborious to be applicable to any 
large number. 

16. Some of the greatest anomalies in marking 
arise from the failure to distinguish the princi- 
ples applicable to (a) examinations designed to 
test the progress of a pupil, from the e applicable 

1 Certain examiners assign Greek letters to individual 
answers in the first instance, and then convert them into 
numbers which are added up, so as to give a total for each 
individual. Thus, if the system indicated in the text is used, 
we have sixteen different symbols, to which are attached 
the successive numbers rero to 15. There is probably a 
real psychological difference between this process and what 
might seem to be the equivalent process of asdgning 15 
marks to each answer.. 
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to (6) examinations deigned to test practical 
capacity. Take the case of simple addition, and 
suppose that the candidate can only do two 
addition sums right out of six. If the examiner 
is only testing the progress of the candidate, it 
may be legitimate to allot to him 30 per cent, of 
the marks ; but, if the examination is designed 
to test his power of adding correctly, he should 
receive no marks, since that power has not reached 
the point of practical usefulness. The same kind 
of argument is applicable to all kinds of capacity 
tests : for these there should be no mark allotted 
in examinations in category (b) above unless the 
candidate obtains considerably more than half 
(say 75 to 90 per cent.) of the marks assigned 
to the particular test ; but the argument does 
not apply to memory tests, in which a candidate 
is legitimately entitled to marks for each item 
remembered. ^ 

17 . In what precedes we have preferred " order 
of proficiency ” to “ order of merit ” because in 
the anonymous examination, as ordinarily con- 
ducted, we know nothing of the “merit,” or 
deservingness, from the iporal point of view, of 
the candidate.' Speaking roughly, proficiency, 
as shown in an examination room, may be said to 
depend inter alia on three variables, (a) natural 
ability, (6) opportunities (including opportunities 
of getting good teaching), and (c) industry and 
perseverance. With a particular purpose in view, 

‘ Se« Not# G, pag# 138. 
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it may be justifiable to |»lace A, who is a clever, 
lazy dog, above B, who is less clever but more 
industrious ; with another purpose in view, the 
order ought to be reversed. In considering a 
permanent appointment to any post, it is clear 
that examination results taken by themselves 
are insufficient. 

When the results of a competitive examinartion 
can be made to depend in part on the report of 
trustworthy persons familiar with the previous 
history of the candidates some account may be 
taken of the relative importance of the three 
factors (a), (b), and (c), in contributing to the 
proficiency shown, and weight may be given to 
each factor corresponding to the general purpose 
of the competition. 

VII. Problem of the Examination-sample 
See criteria (b), (c), and (e), §§ S and 10 above 

18. Examinations may be roughly divided into 
two classes, according as (i) the examination itself 
covers the whole field to be tested and the results 
give complete information as to the character- 
istics required,* or (ii) the field of characteristics 
can only be tested by “ sample." We can show 
with sufficient completeness by examination that 
a candidate has the power accurately to add, 
divide, or multiply, to read a thennometer, or 

* To simplify the problem, we neglect here the variability 
of condition of the candidate (see $ 8S below). 
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measure an electrica]| resistance, the power to 
write in his mother-tongue clearly, grammati- 
cally, and in accordance with a logical plan, to 
speak a foreign language, to write shorthand, and 
a great many other things of primary practical 
importance.' 

If the examination be sufl&ciently complex we 
cap, as in the case of actuaries, cover the whole 
field of a professional examination, so as to be 
able to say with certainty, this candidate can do 
everything required of an actuary. But the 
majority of examinations test only by sample. 
As a typical example of cases where judgment 
by sample fails, we may quote that of an examina- 
tion in ” unseen Latin " in which the examiner 
happens to pitch on passages with which the 
candidate is acquainted. The conclusion drawn 
in such a case is obviously wrong. Even such 
complex professional examinations as those in 
medicine cover only a small portion of the field. 

19. The question of “ fairness of sample ” only 
approaches to definiteness when the field is 
clearly defined either by the well-known require- 
ments of a profession qr by examination syllabus. 
Otherwise the field itself will vary necessarily 
with the examiner, and the comparative success 

' The fact of failure to pass such an examination does 
not necessarily prove that a candidate would not show the 
characteristic tested, if he were placed under conditions 
other than those fii the actual examination (see § SS 
below). 

F fi*? 



THEORY OF EXAMINATIONS 

of two candidates equallylwell equipped in every 
way will depend on whether their views (or those 
of their teachers) of what is important coincide 
or not with those of the examiners. 

20. Let us consider, first, professional examina- 
tions by sample. The professional examinations 
which the public probably regard as of the greatest 
importance are those in medicine. Here ihe 
field of minimum requirements, though constantly 
var 3 dng, is fairly well defined at any given time. 
In order to make sure that a candidate shall not 
succeed by chance, the public insist that he 
shall have gone through a definite course of 
instruction certified by his teachers, in addition 
to passing the examination conducted either by 
his teachers, or by " external examiners,” or by 
a combination of the two. In most other cases 
public opinion is content with less stringent 
guarantees to provide against t^ possibility of 
success by chance, and to ensure that the ex- 
amination-sample shall be a fair sample of 
capacity and knowledge. 

21. in the case of non-professional examina- 
tions by sample, e.g. honours examinations in 
histoiy or literature, the element of chance is 
greatly increased, especially when the examiners 
have no knowledge of the previous training of 
the candidate ; and where such examinations ara 
competitive the element of chance would appear 
seriously to affect the validity of the result (^. 
§18 above). 
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82 .. The question how jiar a particular candidate 
is in a normal condition during the examination 
is a special aspect of the problem of sample. 

(1) The candidate may, by chance, be in an 
abnormal condition, due to temporary indis- 
position. It is impossible in most examinations 
to make any correction to allow for this ; yet 
failure in the case of a candidate who is ill at the 
time of the examination gives no evidence as to 
his powers when he is well. 

(2) We know that capacity-characteristics are 
far more permanent than memory-character- 
istics ; they imply habits of thought, not easily 
destroyed or lapsing with time. From one point 
of view memoryrcharacteristics may be regarded 
as evidence of the capacity to memorise, but the 
ordinary examination gives no information as to 
the time which has been required to memorise 
in a particular r/ise. It is only in the case of 
examinations of extreme difficulty, like the Class 
I, Civil Service examinations, with an age limit, 
and covering a very wide field, that we get valu- 
able evidence of the power to memorise rapidly. 

(3) The candidate may be in an abnormal 
condition due to the examination itself. Some 
candidates are stimulated by the excitement of 
examinations to a display of powers of which they 
are not normally capable ; with the majority, 
owing to nervousness and fatigue, the reverse is 
probably true. 

The question of fatigue is one which demands 
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investigation. Does, foil instance, a written ex- 
amination conducted for six hours a day on five 
successive days diminish the average mental 
powers sensibly, and, if so, does it diminish them 
in approximately the same proportion for all 
candidates ? Do we definitely wish to eliminate 
certain candidates by this fatigue-test ? Will 
the results in subjects requiring special concen- 
tration, e.g. mathematics, be affected according 
as the papers in those subjects are placed at the 
beginning or the end of such an examination ? 

(4) The question of " nervousness ” specially 
affects viva-voce and practical examinations. The 
question whether allowance should be made for 
such nervousness at a viva-voce depends on the 
purpose of the examination. If it is definitely 
intended to test the candidate's power of " think- 
ing on his feet ” before an audience, no allow- 
ance should be made. If it is,jntended only as 
a test of knowledge or capacity in a special sub- 
ject which will not be tested in the same way 
in the career for which the candidate is intended, 
the test itself may be a failure, if the candidate 
is nervous.* In many foreign examinations the 
viva-voce is definitely used as a test of power 6f 
exposition. The case of the practical examina- 
tion is somewhat different, as practical tests have 

1 When tbe purpose of an examination is properly defined, 
it Is easy to show in certain cases (as when it leads to the 
selection or passing of an unsuitable candidate) that the 
examination has failed. 
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rarely to be carried out pelore a critical audience, 
and it is expedient if possible so to conduct 
practical examinations that a candidate shall 
not be able to fail entirely through a single act 
due to nervousness. 

128 . Testing of Schools by Sample. — A special 
aspect of the problem of sample is presented when 
the efficiency of a school is judged by testing a 
certain proportion of its pupils selected either at 
random, or better, from a classified list drawn up 
by the school. The testing, by an external ex- 
aminer, of a few pupils classified by the teachers 
as the best, the average, and the worst may be 
made to yield a fair comparison of his own judg- 
ment with that of the teachers. 

VIII. Influence of Examinations on 
Education and Teaching 

24 . Li 1889 the Hon. Auberon Herbert pub- 
lished a collection of opinions of distinguished 
people entitled. The Sacrifice of Education to Ex- 
aminations, and Mr. Stanley Leathes has prob- 
ably expressed what is still the popular view in 
regarding examinations as a "necessary evil” 
{loc. cit., p. 141). But, as a matter of fact, the 
influence of the examination on the candidate, 
whether for good or bad, depends entirely on 
whether it is well or badly designed for its pur- 
jpose, and the general question " Are examina- 
tions ' good * or ‘ bad * ? " is indeterminate, 

6q 



i.e. it cannot be answere^ in the negative or the 
affirmative. If it is essential for a candidate’s 
future that he shall be able to speak French to 
strangers at a given age, his education up to that 
age will not be sensibly influenced by the fact 
that he is required to pass a viva-voce examina- 
tion in French ; similarly with the examination 
for actuaries (see supra). Here the influence of 
the examination, considered by itself, is nil. The 
nature of the examination and the previous 
education are alike determined by a consensus of 
opinion, and the examination qua examination 
does not influence the candidate or his teachers 
for good or for evil. The fact remains that in 
many cases there can be no such consensus of 
opinion ; and in these cases the examination may 
exert an independent effect, prejudicial or bene- 
ficial, or a combination of both, on previous 
education. 

w 

It is in the nature of things, especially in non- 
professional examinations, that there should be 
great diversity of opinion both as to the proper 
field, and even the purpose, of a particular ex- 
amination, and that diversity will become most 
marked in those branches where progress is being 
made and recent developments appear in rapidly 
changing perspective. The purpose of an exa- 
mination being defined, it is open to question, 
in any given case where there is disagreement 
whether the view of a particular examining body 
or whether the view of a particular teacher is 
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best adapted to the fulfilment of that purpose. ‘ 
But it is certain that where there is such conflict 
of view either the candidate’s teaching must be 
subordinated to the views of the examining body, 
or he must risk failure merely because he is being 
trained with one purpose in view and tested with 
another. If the teacher examines or has a suffi- 
cient share in the design of the examination the 
alternative of conflict or subordination does not 
arise. Where it does arise the personal efficiency 
of the teacher is bound to be diminished if he is 
forced to teach in a way which he regards as 
second best. Professor T. P. Nunn has pointed 
out, on the other hand, that examinations serve in 
certain cases to stimulate teachers, but those 
cases are limited to certain categories. It is 
clear that where teachers have some share in the 
examination of their students the examination 
syllabus is moi^ flexible and can be more readily 
changed from year to year. (For question of 
anonymity and fairness of examinations conducted 
by teachers see § 29, below). 

26. Effect of Examinations on Powers of Exposi- 
tion . — In ordinary life a sensible person only 
speaks or writes on any subject for the benefit of 
a person who knows less than himself; the 

* The indiep«nd«at judgment of the candidate himself, 
Importaoit in questions of culture, is only in the rarest cases 
ihely to be equal or superior to that of examiners or teachers 
.tt other matters, and the sacrifice of the candidate's own 
judgment in training for examinations is generally inevitab>e 
;sea i 87). 
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e'xamination candidate writes for a person who, in 
general, already knows what he has to say, i.e. 
the examiner, and hence may be trained un- 
wittingly to express himself obscurely and by 
allusion. It is of great practical importance to 
counteract this powerful tendency by rational 
training in composition.* 

Sir Napier Shaw thinks that the actual form 
of the three-hours’ papers now set has exercised 
an adverse influence on the teaching given in 
physical science.* 

26 . Effect on Powers of Concentration . — In the 
case of many candidates, examinations, through 
the fear of failure, induce concentration during 
working-time which it is difficult to secure in any 
other way. 

^ Cf. The Writing of English, by the present writer, 
PP» 73> 74 passim. 

* " An evil microbe has got hold of the tegching of physical 
science, for I know of my own knowledge tiat some at least 
of the examiners outside the examination-room take a real 
interest in things, I attribute the sawdust with which the 
examinations are studed to the inhuence of [a] prolonged 
succession of examinations of the same type, the three hours' 
written form of exercise which corresponds with no problem 
that any man or woman has to face in practical life, except 
the sin^e one of keeping awake while the examination lasts« 
"Why can one not invent some other form of examination, 
one, for example, in which candidates are asked, as they are 
in practical life, to take up a job and say when they have 
finished it ? " — The Loch of Science in Modern Education, 
by Sir Napier Shaw, Sc.D,, F.R.S., Director of the Meteoro* 
logical Office (Lamiey Sc. Co., 1916), pp. 13, 14. The pamphlet 
is in part a plea for the study of real things/^ as opposed 
to " laboratory science." 


72 



THEORY OF EXAMINATIONS 

27. The Effect on Edv^ation of Examinations de- 
signed to test Culture . — If culture be distinguished 
clearly from erudition, and be interpreted as an 
individual sensitiveness and responsiveness in 
regard to those things which can be neither 
weighed nor measured, but are nevertheless re- 
garded as of value by the community,* then it 
may well be held that culture cannot be tested by 
any formal examination, but only by social inter- 
course, or by the evidence of original work (thesis 
or dissertation). A written examination can test 
a knowledge of literature, or a knowledge of 
history ; it cannot test the literary or the historical 
sense. But the training for such examinations, 
by checking the individual impulse of the candi- 
date at every turn, may well kill the individual 
culture and originality of effort which it cannot 
test. The sensitive inquiries of the examination 
candidate are likely to be checked at every turn 
by the inevitable question — Will they " pay " at 
the examination ? 

28. The Effect of Examinations on Education by 
Over-absorption of Time . — If examinations cannot 
test culture, still less can they test (except in 
certain limited ways) character, or public spirit. 
It is therefore of the first importance that the 
curricula of candidates, and especially of candi- 
dates in secondary schools, shall be designed so 
that training in the subjects tested by examina- 
tions shall only absorb a portion of the candidate’s 

» See Notes C and D, pp. 93 and 95 bdow. 
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time. In exceptional cases the effect of prepara- 
tion for examinations is *to ruin a candidate's 
health by over-work. The effect of examinations 
on the health of girls and the precautions necessary 
to avoid overstrain in their case deserve special 
consideration. 

IX. Impartiality and Fairness of Exami- 
nations 

29. As applied to examinations, the terms 
impartiality and fairness are not synonymous. 
In cases where the results of examinations are com- 
petitive, it is of the highest importance that they 
should be, as far as possible, not only impartial, 
but fair in the sense that the candidates shall be 
really arranged in order of suitability for the 
purpose for which the examination is held. 

Examinations in which the candidates are 
entirely unknown to the examiners are necessarily 
impartial, but the impartiality may be to a con- 
siderable extent that of chance, and have all the 
injustice of chance. To have anticipated, or to 
have been coached, in a number of memory-tests 
which are later selected by the examiners may 
bring success, but it affords no proof of superior 
suitability. It is to be pointed out that anony- 
mity can only be fully preserved when an examina- 
tion is wholly written ; as soon as the candidate 
is brought face to face with the examine^ at a 
practical or viva-wee examination, the anonymity 
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may disappear. The more searching and personal 
the test, the greater W the possibilities both of 
fairness on the one hand and of partiality on the 
other, everything depending on the character and 
capacity of the examiners. 

Possibly the most satisfactory method of selec- 
tion is to make a first choice anonymously, by a 
written examination, and then to test the surviv- 
ing candidates by viva-voce examination before a 
Board, either with or without evidence from those 
previously acquainted with the candidates. Such 
a Board may of course be prejudiced against 
particular candidates and categories of candidates, 
and against such prejudice there is no safeguard 
but public opinion. As an example of successful 
selection by boards of examiners may be quoted 
the method used by Lord Cromer in choosing 
ofiicers for the Soudan.* The efficiency of such 
methods of selection is tested by the results. 

X. Class I. Civil Service Examinations* 

80. These competitive examinations are of a 
peculiar type. The candidates may select, with 
certain limitations, as many subjects as they like 
out of a large number (to which are allotted 
difiereUt maxima), as long as the total number 
of marks obtainable does not exceed a certain 

* See |>. 45 above. 

* See also Introductioa, p. xii, above, and Note A, 
p. 8 o, below. 
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limit, and there is no necessary relation between 
the subjects which a candidate takes and his 
future career. It is clear that this is a case ir 
which the purpose of the examination is not to test 
directly special knowledge of any kind, but to test 
the power to master a number of subjects ; the 
severity of the competition makes the prepara- 
tion for the examination to some extent a test of 
the candidate's character, inasmuch as to ensure 
success he must for a time sacrifice all other 
interests to it. Success at the examination i? 
followed by different requirements according tc 
the service entered by the candidate. Those 
selected for the Home Civil Service go through a 
period of probation without further examination * 
those selected for the Indian Civil and certain 
Colonial services have to undergo further pro- 
bationary examinations in special subjects re- 
quired for their official work. « 

A fundamental criticism of the methods 
adopted at these examinations is that it is impos- 
sible to compare the suitability of the candidates 
by means of heterogeneous aggregates of marks 
assigned by the different examiners in different 
subjects. Since on the whole the system has 
worked well it may be assumed that the number 
of suitable candidates is considerably in excess 
of the number selected in each year. It is com- 
monly stated that marks may be " more easily 
earned " in subject A than in subject B, but the 
validity of such a statement in respect of different 
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pairs of subjects will almost necessarily vary from 
year to year. 

XL Suggested Use of Examinations to test 

Common Sense and Power of Judgment 

31. Hitherto comparatively little use has been 
made of examinations to test common sense and 
power of judgment in a way for which skilful 
preparation could not furnish any illusory sub- 
stitute. The writer has suggested that " English " 
papers may be set of a kind to test " common 
sense, intellectual tact, readiness, and order.” ‘ 

XII. Experimental Psychological Tests 

33. A large amount of important work has been 
done recently by psychologists with a view to 
determining the natural capacities of individual 
” educands ” by experimental methods. Prof. C. 
Spearman aifti Dr. F. H. Hayward have sug- 
gested that experimental methods may replace or 
coalesce with the examination methods hitherto 
used. In America experimental methods have 
been successfully applied by F. W. Taylor, Mun- 
sterberg, and others to determine relative fit- 
ness for various kinds of industrial work. The 
methods seem capable of great development; 

^ Th* Writing of English, pp. 81-4, and paper on Examinair 
tlons, read before the British Association, published in the 
School World, October 1911 ; the new English papers pro- 
posed by Mr. Stanley Leatbes' Committee for Class 1 . Civil 
Service Exaodnations sbould serve this purpose (see p. 9s), 
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and a system of tests is being used in American 
“ vocational bureaus ” for tAe purpose of advising 
youths in the selection of a career. It is possible 
that the whole system of examinations may be 
ultimately transformed by the use of these 
quantitative methods; but the matter is stiU in 
a rudimentary stage. 

XIII. Medical Tests 

83. For certain careers a medical test of fitness 
is essential. There is a movement on foot to 
extend medical tests and to give marks in Univer- 
sity degree examinations for physical capacity 
as shown in Swedish drill and the g 5 nnnasiiun, a 
return to Greek ideals. 

XIV. Conduct of Examinations 

« 

84. To avoid chance hardships many precau- 
tions of detail are necessary in the conduct of 
examinations, of which the following are a few 
examples, taken at random as illustrations. 

(i) It is the practice of some examiners to 
attribute different mark-values to different ques- 
tions, leaving it as an additional test of the candi- 
date’s intelligence to determine in what way they 
can earn the largest number of marks within the 
period of the examination. Definite instructions 
on this point and on the minimum number of 
answers which must be attempted to ensure a 
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pass should be given if the examining body 
holds a particular vieW on the matter. 

(2) In the case of mathematical papers the 
examiners should work out the questions in full, 
to avoid unexpected difficulties and even " traps.” 

(3) It is essential that examiners should not be 
required to mark an excessive number of scripts 
in a given time. (See p. no.) 

XV. Conclusion 

35 . In what precedes a method has been 
sketched out and illustrated, for dealing with 
examination problems. But the question is only 
in its infancy. A systematic investigation (in- 
cluding statistical investigation) of the method 
and results of examinations is urgently needed. 
A great advance would be effected if examining 
bodies could be induced to express in words (1) 
the purposes o^ their various examinations, and 
(2) as exactly as possible what the ” passing ” of 
a particular examination means in terms of what 
all the successful candidates could certainly do 
at the time of the examination — in other words, 
non-professional examihations should be given 
the same precise significance as professional 
exanunations. 
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EXTRACTS FROM THE FOURTH REPORT OF 
THE ROYAL COMMISSION ON THE CIVIL 
SERVICE (MAJORITY REPORT). 1914.1 

(i) First two paragraphs of Section 7 (p. 29) 

7. But before proceeding to discuss in any detail 
the schemes of examination by which we propose 
that those tests should be carried out, we desire to 
emphasise our opinion that an examination scheme 
is only part, and not the most important part, of the 
machinery by which the State should aim at the selec- 
tion of suitable candidates for its service. 

Unless the general educational system of the country 
is efficient, the natural powers of 'those who would 
make the best candidates will neither be discovered 
nor trained. 

{2) Third paragraph of SeUion 8, and first two para- 
graphs of Section 9 (p. 31) 

8. In Chapter I we have indicated the origin and 
followed the development of the competitive system 
of recruitment for the Civil Service. With the pro- 
gress of this development the efficiency of the Service 

1 [Cd. 7338.] Published by the Stationery Office. Rice 
IS. 4^., to be purchased of any bookseller. 
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has grown until at the present time the State possesses 
a body of public officers jvho are far more competent 
and zealous than their predecessors, appointed under 
the regime of patronage, are stated on official autho- 
rity to have been. We have no doubt whatever that 
to this highly satisfactory result the system of com- 
petitive examination has mainly contributed. The 
system has, in our opinion, entirely justified the 
expectations of its originators. It is true that it 
never has been, and is not now, exempt from 
hostile criticism, and we do not affirm that the 
written examination is an infallible or a final test 
of the best results of education. But the defects 
attaching to the system, which we shall in due place 
examine, are, perhaps, susceptible of some mitiga- 
tion. In existing political conditions, and in such 
developments of them as can be reasonably antici- 
pated, we believe that the advantages of the system 
of competitive examination as a means of recruitment 
for the Civil Service far outweigh any defects which 
have come to o\jr notice, and we are convinced of the 
importance not only of adhering to the system, but 
of extending it whenever possible. 

9. Since, then, examination should, in our opinion, 
play so large a part in the selection of this most 
important body of officials, it becomes of supreme 
importance so to order and arrange the schemes and 
improve the methods of examination as to secure 
the best results. 

How far the existing arrangements secure this will 
appear from a review of the schemes of examination 
by which at present access is gained to the different 
administrative and clerical classes. We propose 
G 81: 
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now to undertake such a review, and to recommend 
such changes as appear desirable, 

(3) Sections 26 and 27 (pp. 34, 35) 

26. We have already drawn attention to what is, 
in our judgment, a serious defect in the existing 
arrangements of the examinations for entry into 
the clerical branch of the Civil Service, viz. their 
divergence from the actual conditions of education. 
The Second Division examination marks the extreme 
point of such divergence. 

The three years seventeen to twenty correspond to 
no stage in the organisation of education in this 
country. As a complete course of secondary educa- 
tion at present ends when the pupil is about eighteen 
years of age, it is obvious that, while the inferior age 
limit is now too low, the superior limit is far too high. 

27. An inevitable, and in our judgment undesir- 
able, consequence is the prevalence of special pre- 
paration ” or “ cramming ” for this examination. 

We have taken pains to ascertain the extent to 
which cramming prevails, and we find that more 
than half the candidates undergo some kind of special 
preparation for this examination. To some extent 
recourse to the " crammer ” is explained by the 
circumstances of boys already in employment during 
the day who have only the evening for study, and 
for whose needs no other agencies suffice. We attach 
greater significance, therefore, to the statistics show- 
ing, for candidates who give their whole time to 
instruction, the percentage who attend places of 

special preparation." 

Such statistics show that the number of candidates 
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who leave school for this purpose is greater between 
15 and 16 than between 14 and 15 years of age ; 
greater still between 16 and vj, and greatest between 
17 and 20, 

This tendency is doubly objectionable. In the 
first place such special preparation necessarily costs 
money, while those boys from the elementary schools 
who are likely to be successful in this examination 
would, in most cases, be receiving gratuitous instruc- 
tion in the public secondary schools. Moreover, in 
so far as this system makes success dependent upon 
an expensive substitute for the ordinary school educa- 
tion, it necessarily places the poorer candidates at a 
disadvantage. 

In the second place, the interruption of a school 
curriculum is injurious to the boy’s best interests. 

More general recognition is now being won by the 
great principle that the most valuable elements in 
education are those formative influences which help 
to mould and develop character. Such influences 
form no part oi»the atmosphere of a " cramming ” 
institution, the main object of which is to surmount 
or evade the difi&culties of an examination paper with 
the smallest expenditure of time and trouble. 

We do not deny that useful instruction may be 
given in such institutions. In so far as they supple- 
ment the strictly educational curricula of the schools 
of the country by teaching such technical subjects 
as book-keeping, shorthand, and typewriting, they 
perform a useful service to the public ; but their 
professed aim is different from the aim of true educa- 
tion* The principal aim to be kept in view in re- 
cruiting the Civil Service (towards which examuxa- 



EXAMINATIONS— NOTE A 


tions are a necessary, if a somewhat fallible means) 
is to obtain the best results f>i the regular educational 
system which is in actual operation ; and, when that 
system becomes sufficiently uniform throughout the 
United Kingdom to permit such a course, we think 
that 'Government should require a school-leaving 
certificate as a condition of candidature for the clerical 
examinations. 

(4) First two lines of the fourteenth paragraph of Section 

43 (P- 40) 

43. It is to be remembered that competitive ex- 
amination is a means, though necessarily an imper- 
fect one, of testing natural ability, as well as acquired 
training. 


(5) Section 44 (pp. 40, 41) 

44. In regard to the suitability of the present 
Class I examination, academic opinion was not 
unanimous. There was, indeed, something approach- 
ing agreement in the view that the competitive 
examination is the most, if not the only, practicable 
means of selecting candidates for the situations which 
we are now considering. But the imperfections of a 
written examination as a test, not merely of know- 
ledge, but also of capacity — of the faculty of sound 
judgment, common sense, resourcefulness, and reso- 
lution — ^were often insisted upon. 

To make good these defects several suggestions 
have been put before us, to two of which in particular 
we invite attention, viz. : (a) the selection by the 
authorities of the various Universities of Honours 
men, also distinguished for their judgment, good sense, 
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and general capacity in student life, for competition 
inter se for the situationsvacant ; and (b) the institu- 
tion of a viva-voce examination supplementary to the 
written test, with the object of ascertaining the can- 
didate's general mental calibre, and the possession 
of those qualities of common sense, sound judgment, 
resourcefulness and resolution, upon which the 
wrijten paper rarely gives assurance. 

The Vice-Chancellors of the Universities of London 
and Wales entertained a favourable opinion of the 
former of these suggestions ; indeed they seemed to 
regard it as likely to become the normal method 
of recruitment of the future. We have considered 
whether it could be applied as an experimental 
measure to a proportion of vacancies annually, but 
we are satisfied that the practical difficulties are too 
serious to permit even a partial or tentative trial of 
the system. We are unanimously of opinion that, 
whatever the future may have in store for us in this 
connection, the method cannot generally be applied 
to-day, if only iti view of the impossibility of closing 
the competition to non-university candidates. 

(6) First paragraph, and first eight lines of the second 
paragraph of Section 45 {p. 41) 

45. In respect of the second suggestion referred to 
above, we call attention to the opinion which the 
Civil Service Commission entertained on this matter 
in the earlier years of their activity. In their Tenth 
Report (1865) they say of the examination for the 
Indian Civil Service : — In each subject there is a 
viva-voce examination, to which we attach great im- 
portance, introduced for the express purpose of fully 
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testing the genuine character of a candidate's know- 
ledge, in addition to calling into play those qualities 
which are little, if at all, elicited in an examination 
by written papers, viz. his readiness, self-reliance, 
and moral courage.” We have ascertained from the 
Civil Service Commission that this practice of viva- 
voce examination was continued until the year 1891, 
when it was abandoned, apparently owing to. the 
increase in the number of candidates caused by the 
raising of the age limits. 

We do not now propose for the Home Service the 
establishment of the viva-voce examination conducted 
by each examiner in each subject taken by a candi- 
date, although some of us believe that such an exami- 
nation would be an effective help towards determining 
the candidate’s knowledge of the subject. Our 
object is different : it is to ascertain whether a general 
viva-voce examination based on the whole syllabus, 
and conducted by a single committee of three care- 
fully chosen examiners, might not be made an effec- 
tive help towards determining tho^'e of the candi- 
date’s qualities which are little, if at all, tested in an 
examination by written papers. 

% 

(7) Section 46 (p. 41) 

46. Certain academic witnesses seemed to think 
that the present syllabus gives undue weight to the 
studies most commonly followed in the older Univer- 
sities and that the modem institutions are handi- 
capped by it in the competition. Undoubtedly the 
most salient fact emerging from an analysis of the 
academic antecedents of the successful candidates 
is the great numerical preponderance of those who 
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have been educated at Oxford and Cambridge, especi- 
ally at Oxford. . • 

(8) Section 50 (pp. 42, 43) 

50. However conclusive in the opinion of some, 
and probable in the opinion of others, may be these 
explanations of the facts mentioned in paragraph 46 
above, they do not in our judgment offer a sufficient 
reply to the inquiry whether the conditions of the 
examination, the grouping of the subjects, and the 
relative values assigned to each group, do or do not 
give to any subject or course of study an undue 
advantage in the examination hall. ... It is, in our 
opinion, highly undesirable that such an inquiry 
should remain unanswered. A satisfactory answer to it 
can be given in only one way, namely, by a full inves- 
tigation into the facts. We were glad to learn from 
the Civil Service Commissioners themselves that they 
concur in the necessity for such an inquiry ; and we 
accordingly recommend the Government to appoint 
a Committee with the object of ascertaining whether 
there is any foundation for or reality in the misgiving 
which certainly prevails in some quarters that the 
scheme of examination for the Class I. unduly favours 
the curricula of the older Universities and handicaps 
those of the newer. On the Committee the Civil 
Service Commission and both University and School 
experience should be represented, and it should be 
presided over by some man of recognised judgment 
and eminence in public life. Should it be found that 
any change is desirable, then it would be necessary 
for the Committee, while maintaining the high stan- 
dard necessary for the examination, to revise and 
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rearrange the syllabus, weighing, with even balance, 
the educational values of the studies of the older 
Universities against those of the modem learning, as 
it has been called, to which the younger Universities 
are thought to pay more attention. The revised 
syllabus thus made out should be submitted to the 
Government with whom the decision must rest. 

f 

(9) Sections 4 and 5 of the Summary of Recommenda- 
tions (p. lOl) 

The General Civil Service, Administrative and 
Clerical. 

(a) General 

4. The competitive examinations for recmiting each 
class of officer, administrative and clerical, should be 
adjusted in respect of the age of competitors and the 
subjects of competition to the stages of the educational 
system actually existing in the country. (Chapter 
III, § 6, p. 29.) 

5. The examinations should be directed to testing 
the natural ability of candidates, and the results of 
their education both with respect to acquirement of 
knowledge and the formation of mind and character.^ 
It should not be direeted to testing proficiency in 
particular subjects which lie outside the normal scope 
of education. 

* The Commission give no hint to the Civil Service Com- 
mission as to how examinations are to be directed to testing 
the formation of character. — P, J. H, 
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ON THE REPORT OF THE TREASURY COM- 
MITTEE ON CIVIL SERVICE, CLASS I. 
EXAMINATION (1917) 1 

The Committee referred to in the heading was con- 
stituted as follows : Mr. Stanley Leathes, C.B, 
(Chairman) ; Sir Alfred Ewing, K.C.B., F.R.S. ; Sir 
Henry Miers, D.Sc., F.R.S. ; Mr. H. A. L. Fisher, 
F.B.A. (replaced on his appointment as President 
of the Board of Education by Dr. W. H. Hadow, 
M.A.) ; and Prof. W. G. S. Adams, M.A., with 
Mr. D. B. Mair, M.A., Director of Examinations to 
the Civil Service Commissioners, as Secretary. The 
reference to th% Committee was : 

To consider and report upon the existing scheme of 
examination for Class I. of the Home Civil Service : 

To submit for the consideration of the Lords Com- 
missioners of His Majesty’s Treasury a revised scheme 
such as they may judge to be best adapted for the 
selection of the type of officer required for that class 
of the Civil Service, and at the same time most advan- 
tageous to the higher education of this country : 

And in framing such a scheme, to take into account, 
so far as possible, the various other purposes which 
the scheme in question has hitherto served, and to 

‘ [Cd. 8657.] Price 3d. Published by H.M. Stationery 
Office, and to be purchased of any bookseller. 
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consult the India Office, the Foreign Office, and the 
Colonial Office as to their requirements, in so far as 
they differ from those of the Home Civil Service. 

The scheme of the Committee may be summarised 
briefly as follows ; 

The examination is to be divided into three 
sections : Section A, an examination mainly general 
in character, obligatory for all candidates, with a 
maximum of 800 marks ; Section B, a specialised ex- 
amination, including fifty-nine separate subjects with 
maxima varying from 100 to 400, candidates being 
allowed to take up subjects in the section up to a 
total of 1,000 marks ; Extra Numerum Section, cover- 
ing 100 additional marks for translation papers in a 
language not already taken by the candidate. 

The main points in which the scheme differs from 
the existing scheme (September 1917) is in the char- 
acter of Section A, and the readjustment of the scale 
of marks in Section B, which is intended to place 
on an equality “ the chief studies which are pur- 
sued by students at the University up*to an Honours 
Course, viz. Classical languages, history and litera- 
ture ; Modem languages, with history and literature ; 
History ; Mathematics ; and the Natural Sciences.^ 

Section A includes the following items ; (i) Essay 

(100 marks) ; (2) English (100 marks) ; (3) Questions 
on contemporary subjects, social, economic, and 
political (100 marks) ; (4) Questions on general 

principles, methods, and applications of Science (100 
marks) ; (5) Translation from one of a number of 

^ To obt^n the maximum in Natural Sciences, candidates 
must take one main subject up to the '* higher level " and 
two subsidiary subjects up to the lower level. 
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European languages, or Latin (lOo marks) ; (6) A 
voce examination (300 marks). Total, 800 miarks. 

“This Section,’’ says the Report, " is intended to 
test the candidates’ knowledge of the English language 
and their capacity for its skilful use, their accurate 
command of knowledge which they should have 
acquired in the course of a systematic education and 
self-education and should have retained to assist 
them in their future work, and their equipment in 
one foreign language at least for working purposes. 
The languages selected are those most likely to afford 
information useful to public servants. As circum- 
stances change others should be added at the dis- 
cretion of the Civil Service Commissioners. 

"The viva voce should be a test, by means of ques- 
tions and conversation on matters of general interest, 
of the candidate’s alertness, intelligence, and intel- 
lectual outlook, his personal qualities of mind, and 
mental equipment.* 

“ It is not intended that any candidate should be 
disqualified f6r failure in any of the parts of this 
Section, or in the Section as a whole, but that the 
Sect ion should count substant ially in the com pet it ion . ’ ’ 

The Section is devised to ensure, as far as possible, 
that knowledge of both science and the humani- 

ties which Huxley, a pioneer in this matter, pointed 
out as essential for every " well-educated man.” 

Before Huxley’stime, education meant the humani- 
ties only ; since his time some would have it mean 
" science ” only, in a narrowly restricted sense. If 

* [C/., on this point, section 45 of the Majority Report of 
the Royal Commission on the Civil Service ,19*4. quoted on 
p. 83 above.] 
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the Committee’s scheme is adopted, the Class I. 
examinations will at any rate be free from the reproach 
that they tend to perpetuate the " neglect of science ” 
which has been so serious a reproach to our educa- 
tion, and which has had such grave consequences 
in the War. The Committee have given specimen 
examination papers ; but the principles to be 
adopted in marking are only vaguely indicated by 
the following rubric : “ Each candidate’s answers 
will be valued as a whole, and any candidate who 
presents answers to less than three or more than four 
of the questions will be presumed to do so in knowledge 
of the value of the answers which he is able to give 
to the questions he chooses.” 

The specimen English paper, which is on lines 
largely in accordance with suggestions put forward 
at various times by the present writer, constitutes 
a new departure in examinations of this character. 
It is a test of general intelligence of which the result 
cannot possibly be falsified by special preparation. 

The re-introduction of the viva-voce ffest, to which 
300 marks are attached, would again, as I have sug- 
gested (pp. 21-22), constitute a real reform. But it is 
one which needs very careful safeguarding ; and I 
am not sure that it would not be well to make it 
public, like so many foreign viva-voc- examinations. 
The presence of strangers who could not influence 
the verdict would add little or nothing to the severity 
of the test, for which clearly the candidates would 
prepare themselves, by undergoing similar tests in 
class before their comrades.* 

* Cf. " The Aim of the New Curricula of French Secondary 
Schools for Boys,” by the present writer, in Special Reports 
of the Board of Education, vol. 24, 
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THE CONNOTATION OF THE WORD " CUI^ 
" TURE " (see also Note D) 

There are many interesting passages bearing on this 
point in J. H, Newman’s The Idea of a University, 
third edition (1873), especially in the “ Discourse 
on Knowledge viewed in relation to Learning.”^ 
The following (p. 135) is typical of Newman’s views ; 

“ When this analytical, distributive, harmon- 
izing process is away, the mind experiences no 
enlargement and is not reckoned as enlightened 
or comprehensive, whatever it may add to its 
knowledge.” 

Newman seems to have in his mind only that colder 
aspect of additions to knowledge by assimilation of 
fresh facts to pre-existing intellectual possessions 
which the modern psychologists call “ apperception.” 

The following passage from Goethe gives a fuller 
description of the same process, in which the emo- 
tional aspects are taken into account ; 

* The book was originally published in 1852 under the title. 
Discourses on the Scope and Nature of University Education, 
etc., and the particular essay referred to was entitled, "On 
Philosophical Knowledge viewed in relation to Mental 
Acquirements." 
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' V Everything of outstanding excellence {Alles 
Vorireffliche) gives us for,the moment a sense of 
limitation, because we feel we have not reached 
its level ; only in so far as we afterwards make 
it part of our own culture [Cultu r) and an element 
in our own mind and spirit does it become to us 
dear and valued." ^ 

There is an interesting parallel passage in E. T, 
Benson’s clever contemporary novel, The Climber : 

"... but that assimilation which is necessary 
before facts can become the food of culture, 
that kindling of the blood, as with romance, had 
not occurred." 

I do not quote from Matthew Arnold’s Culture 
and Anarchy, partly because it is so well known, 
partly because I propose to deal with some aspects 
of it elsewhere. 

In the text I have compared the individuality of 
culture with the individuality of cons'cience, a com- 
parison suggested to me, I think, by an interesting 
book by the French physicist, Henri Bouasse (whose 
constructive suggestions seem to me more valuable 
than his criticisms of his fellow-workers), but the 
comparison ought not to be pushed too far. Culture 
is clearly to be distinguished not only from the moral 
but from the " intellectual ” conscience to which I 
have referred in the Writing of English, p. 63. 

^ Maximen und Reflexionen, Goethe Gesellschaft edition, 
1907, p. 65. 
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NOTE D 


KULTUR AND CULTURE 

The history of the word KuUur ^ in Germany is from 
the point of view of pure semantics almost a para- 
dox ; from the point of view of the history of man- 
kind it is grave and tragical — corruptio opiimi pessima. 

Thanks to successive editions of the Dictionary of 
J. C. Adelung, we can safely say that the word 
became of general use in Germany between 1774 and 
1793. It was introduced from the French, and took 
with it the two active significations, the literal one 
of cultivating the land, and the figurative one of culti- 
vating the mind, without any marked departure from 
the French original.* But in the figurative sense in 
which the word is most generally employed, to signify 
the result of such cultivation as applied to man, there 
is a wide and important departure. Whereas " cul- 
ture,” both in French and in English, is always 
used with special empha,sis on the intellectual and 

* Cultur was originally spelt with a C, and is so spelt 
in most of the earlier texts. J. H. Campe, however, in 
the posthumous edition of Adelung’s Dictionary points 
out that the letter C is unnecessary in German, as it may 
always be replaced by K (as in Kultur), or by Z ; and this 
method of spelling is now in general use in Germany. 

* C/. the definition of culture in the edition of the Dictionary 
of the French Academy of 1762. 
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aesthetic aspect of its general connotation (Murray’s 
Dictionary gives as one of the meanings “ the intel- 
lectual side of civilisation ”f, in German the moral 
aspect is inseparable from the use of the word, 
except in the specialist writings of one or two phil- 
osophers.^ Adelung, who, by his use of the word in 
the compound KuUurgeschichte, and, by his popular 
work on that subject, largely contributed to its accept- 
ance as an equivalent for the English “ civilisatioii^’ 
defines the word as meaning " the ennobling or re- 
fining of the whole spiritual and bodily powers of a 
person or a people, so that the word connotes not 
only the enlightenment {Aufkldmng) and ennobling 
of the understanding by delivering it from prejudices, 
but also the polishing (Politur) and refinement of the 
moral qualities {der Sitten).” At the beginning of the 
nineteenth century the word was used, in Germany, 
and it still is used, in the following figurative senses ; 

(i) To signify “ state of civilisation ” in its widest 
sense, without any implication of such a state being 
good or bad. It is this sense that ij; carries in the 
widely used expression KuUurgeschichte. The 1897 
edition of Meyer’s Konversationslexikon states that 
KuUurgeschichte requires an “ investigation of the 
whole social life, including dwellings, hygiene, clothing, 
furniture and utensils, way of living, food, morals 
and customs, legal conceptions, beliefs and super- 
stitions, at different periods.” The distinguished 
English ethnologist, E. B. Tylor, in his book on 
Primitive Culture, uses the word exactly with such 
connotation as a synonym for ” civilisation in its 

* For example, H. Rickert, in his Kuliurwissenschaft und 
Naturwissenschafl (1910}, see pp. 26, 27 and passim, 
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widest sense.” ^ It seems clear that Tylor was 
influenced by the German use of the word 
" Kultur,” and so gave to the English word 
‘‘ culture ” a sense which it did not previously 
possess, and still only possesses in English as a 
term of ethnology. 

(2) Tomcan" civilisation as opposed tobarbarism,” 
t.b.^ " Kultur ” of people who are regarded as 
being on a high plane. Goethe, in a remarkable 
rpassage in the Conversations with Eckermann (third 
part, March 14th, 1830), contrasts Kultur und Bar- 
bar ei.^ The sense of civilisation as opposed to bar- 
barism is also well illustrated by the following passage 
from Schiller. Speaking of the influence of Italy on 
France in the Renaissance of the sixteenth century» 
he says; “The dawn of Cidtnr appeared. France 
was already hastening to welcome her civilisation ” 
{i.e., after the Dark Ages).* 

From this limited sense of civilisation were de- 

1 See Preface to Primitive Culture, 4th edition. The first 
edition appeared'in 1871. 

3 But a little later in the same passage Goethe uses it in the 
first and more general sense, and says : “ There is something 
singular about national hatred ; one always finds it strongest 
and most violent at the lowest stages of civilisation (Kultur) ” 
Goethe is explaining to Eckermann why he declined to write 
songs of hate against France, and asks how he could write 
songs of hate without hating, and how could he, to whom 
culture and barbarism were the only things that mattered, 
hate the French, who were among the most cultivated of 
people, and to whom his own culture (here he uses the word 
Bildung, see footnote on p. 99 below) was so largely due. 

® ** Die Morgenrdthe der Cultur erschien. Schon eilte 
Frankreich mit schnellen Schritten seiner Civilisirung enU 
gegen " (Geschichte der franzdsischen Umuhen, etc., K. 
Goedeke's edition of Schiller's Works, 1867, vol. ix. p. 294). 
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veloped three more specialised uses of the word, 
set out below. , 

(3) Moral elevation. Goethe writes in his Maxi- 
men und Reflexionen (Anhang, Cotta edition, vol. iv. 
p. 230) : “ Piety {Frommigkeit) is not an end in itself, 
but a means by which to attain through the purest 
spiritual quiet to the highest moral elevation {CuUur)." 
We find the same use of the word in Dichtung u^d 
Wahrheit, Part II. Book ix. (Cotta edition, vol. xxiii. 
p. 188). Goethe also sometimes uses the phrase 
“ sittliche Kultur" (see, e.g., Dichtung und Wahrheit, 
Part II., Book vii., loc. cit., p. 9). 

(4) Individual refinement and sensitiveness, com- 
bined with knowledge, in the intellectual and aesthetic 
spheres. This is approximately the sense in which I 
have used " culture ” in the text. It is a rare sense 
in German. An example of its use by Goethe is 
given in Note C, p. 94, above. 

(5) External polish, relating to dress, manners, etc. 
It is in this sense that Goethe uses it in Faust (Part I. 
lines 2495-6), where Mephistopheles explains to the 
Witch his reason for concealing his horns and tail as 
out of fashion. 

“ Auch die Cultur, die alle Welt bedeckt, 

Hat auf den Teufel sich erstreckt.” 

which may be freely rendered : 

" Of culture too, which all good folks now wear. 

The Devil himself has got a share." 

With a word of such wide connotations individual 
differences of usage by different authors would in any 
language be inevitable, but the central fact in the 
history of the word “ Kultur ” in Germany is that 
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the most frequent and common sense in which the 
word is used is that ot civilisation as opposed to 
barbarism, and with stress laid on the moral side of 
civilisation. 1 Absurd and ridiculous as it may seem, 
it is quite clear that it is because of this sense that 
the Pan-Germans in their own country have been 
able to use it as a signal and a watchword almost 
comparable to the Liberty, Equality, and Fraternity 
of the wars of the French Revolution. To us it now 
jiands for the massacre and enslavement of Belgium, 
Serbia, Armenia, the repudiation of all international 

i It has been stated that the proper translation of " cul- 
ture in the English sense is Bildung, and not KtiUur, Bil- 
dung is a synonym of the fourth sense of KuUur, which is 
rare in German, and to which it corresponds almost exactly. 
Thus F. Paulsen says of Bildung that it consists not merely 
in the possession of things known (K$nntnisse) but in the 
possession of the hving forces of active knowledge (des Er- 
kennens) and of action in which the inner living individual 
(die innere Lebensform) takes part (Article on " Das Modern© 
Bildungswesen ip P. Hinneberg's Die Kultur der Gegcn- 
wart (1906). Goethe, however, seems to limit the word 
Bildung rather to a receptive assimilation of knowledge than 
an active one. 

It has also been stated that Zivilisaiion in German means 
only material civilisation. That this is not strictly accurate 
is shown by the quotation from Schiller (on p. 97 above). 
Harnack, the theologian, in a passage written in 1909, says 
that England's '' Zivilisaiion " is, taken as a whole, still 
superior to that of Germany, when, from the argument, it 
is quite clear that he means something much wider than 
material civilisation (A us Wissensekaft und Leben, 1911, 
p. 200). On the other hand, W. Muench regards Kultur as 
consisting of “ internal Kultur** often called Kultur, and 
“ external Kultur,** called ** Zivilisaiion,** the two being the 
inseparable wings of a " single front " (Aus Welt utid Schule, 
1904). 
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law, the atrocities of Louvain and Rheims. The 
moral degradation of Germa'ny has degraded the word 
itself. The degradation has not taken place without a 
long struggle, and the uses and definitions of the word 
Kultur by men of different political schools. Liberal 
and Socialist on the one side, and Pan-Germanists 
on the other, are historically significant of that 
struggle. Harnack contrasts ironically the Kultur 
of the present day, with its arriere-pensees'^ and 
compromises and concealment of its original sourctT, 
with the spirit of scientific investigation (Wissen- 
schaft) which knows no compromises and no arriere- 
pensees, and suggests that, if things had to be settled 
by contemporary authority, Awtor would be entrusted 
to “ highly trained journalists,” while Wissenschaft 
would be shut up in a monastery.* 

Paul Michaelis, a liberal journalist advocating a 
closer union between Germany and England in his 
book on Prussian Culture,* actually introduces into 
his definition of Kultur the phrase ” the will for 
Peace.” But Bernhardi, in his book on the Future 
of Germany, published in 1913,* demonstrates that 
an essential part of Kultur is the will for War, his 


‘ Harnack use.s the word Riichsickt ; I know of no English 
equivalent for the word in this particular sense. 

* Reden und Aulsdtze, 1904,11., p. 371 ; the passage itself 
dates from 1905. Harnack, alas, signed the Chauvinist mani- 
festo of the German professors in 1914 to which the noble 
and moderate reply of the French Universities forms so 
striking a contrast, 

3 Von Bismarck bis Bethmann. Die Politik und Kultur 
Grosspreussens, 1911, p. 271. 

* Unsere Zukunft, see pp. 40-43. The book has not, so 
far as 1 am aware, been translated into English. 
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demonstration being based on the theory that " Ger- 
man Kultur is the most necessary factor in human 
progress,” and that the acquisition of colonies by 
Germany is essential for the propagation of German 
Kultur. “ Kultur ” is the Red Cross flag under 
which the German pirates have sailed to the attack 
on humanity. 

The object of this note has been partly to clear 
up some current misconceptions, but mainly to show 
J;hat our fine word “culture,” which has no synonym 
in English, ought not to be driven from our language 
by the degradation of Kultur. I hope to deal with 
the question more fully elsewhere. 
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NOTE E 


PROF. EDGEWORTH’S INVESTIGATIONS ON 
THE STATISTICS OF EXAMINATIONS 

1. The only published statistical investigations 
importance on the results of examinations of which 
I am aware, arc contained in three memoirs by Prof. 
F. Y. Edgeworth, of the University of Oxford ; 

(i) The Statistics of Examinations, Journal of the 
Royal Statistical Society, vol. li., pp. 599-635 (1888), 

(ii) The Element of Chance in Competitive Examina- 
tions, id., vol. liii., pp. 460-75 and 644-63 (1890), and 

(iii) On Problems in Probabilities {Philosophical 
Magazine for August, i89<').' 

2 . Definition of Standard or True M{irk and general 
Method of Investigation. — In comparing the marks of 
different examiners with one another, it is necessary to 

‘ These papers also contain references to other works by 
the same distinguished author, including a treatise on the 
art of measurement, entitled Metretike {London : Temple Co.). 
An elaborate account of the mathematical theory of Proba- 
bility is given in Professor Edgeworth’s article on that sub- 
ject in the eleventh edition of the KncycloptBdia Bfitannica, 
The first two papers, which are summarised below, are intended 
for the general reader ; the third contains a more technical 
treatment of problems dealt with in the first two. The 
present writer has no expert knowledge of statistics, and is 
greatly indebted to Professor Edgeworth for having revised 
this summary. 
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assume that there is some standard, or true mark. 
In the second paper, Prof. E. postulates '' that the 
true or standard mark of any piece of work is the 
average of the marks given by a large number of 
competent examiners equally proficient in the subject 
and instructed as to the character and purpose of 
the examination."' ^ 

Prof. E. clearly states this as a postulate, not as 
an axiom ; but it is a convenient postulate, and is 
^yrobably generally acceptable. The general method 
adopted by Prof. E. is to ascertain what this standard 
is in a number of typical cases, and to investigate 
the divergency from this standard, or error, of the 
marks allotted by individual examiners, and to deduce 
general laws of such errors, with the object of ascer- 
taining the probability that an error of assigned 
extent will be committed by a competent examiner 
in marking a single piece of work of this kind ; and 
from this to deduce the probable errors in the more 
complex cases which present themselves in practice.^ 

^ This is a de^lopmcnt of the corresponding statement in 
the first paper. 

2 Even at the outset there are at least two practical diffi- 
culties to be got over : (a) the difficulty, involving as a rule 
expense, of obtaining a large number of “ equally competent 
examiners to look over and mark the same piece of work ; 
and (6) the difficulty of obtaining any definition of the ** pur- 
pose " of the examination in question. As I have pointed 
out in the text, this absence of definition of purpose con- 
stitutes, in my judgment, one of the great weaknesses of 
our present system, and one which necessarily leads to far 
greater divergencies of marking than w’ould otherwise exist. 
Prof. E., in his second paper, expressly states that his subject 
is not the philosophy of examination, but the statistics of 
marks,'* and does not attempt therefore to go behind the 
marks allotted. 
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3 . Mutual compensation of independent errors . — 
Effect on aggregates of marks. — Prof. E. points out 
that, according to the Calculus of Probabilities, if a 
certain quantity is liable to a probable error of i per 
cent, in its measurement, the probable error on the 
aggregate of the sum of a hundred such quantities 
is not I percent., but only per cent., that is i per 
cent, divided by the square root of loo, “ in virtue 
of the mutual compensation of independent errors.” 
The statistical theorem of which this is an exampl^ 
is much used throughout his invest igations.^ (See 
also § 24 , p. 130.) 

4 . The above considerations are common to the 
two papers, of which I now submit separate sum- 
maries, necessarily brief and imperfect, but bringing 
out certain salient facts. 

‘ The whole investigation is based on the Mathematical 
Theory of Probabilities. The following definitions and state- 
ments will be useful to the reader : — 

(1) The probable error of a series of observations is that 
divergence from their mean on either side f . . within which 
exactly half the observations lie. An approximation to the 
probable error for a given series of observations is obtained 
by arranging all the observations in order of magnitude; 
marking the magnitude, say, a, above which 25 per cent, 
of the observations lie, and the magnitude, say /9, below 
which 25 per cent, lie. Half the di If ere ace between a and 
/S is the probable error, 

A useful way of illustrating this is to say that if one 
observation is chosen at random out of a group it is as 
likely as not that it will not lie further from the average 
than the probable error. (A. C. Bowley's Elements of 
Statistics, second edition, 1902, pp. 281-2.) 

(2) If n marks (or other magnitudes) are arranged in 

an ascending order of magnitude the mean of the mark 

4 
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Paper I ^ 

5. Choice of an Idgal Typical Examination for 
Investigation . — In order to make the subject more 
readily intelligible, Prof. E. takes as a particular case 


and the (- + i)'** mark is called the lower quartile ; the mean 
4 

■~j and the (-+1)^** marks is called the mean of 
the set; and the mean of the and marks 

is called the upper quartile. The probable error is thus half 
the difference between the upper and lower quartiles. 

It will be seen that the “ possible error is always con- 
siderably in excess of the probable error. 

(3) The mean error of a series of n observations of 
which the individual errors or deferences from the mean 
value are ex, es . . , is the square root of the sum of 
the squares of those errors divided by their number, i,e,, 




ei^ -f ^3’ 4 “ ^3* 




For aggregates fluctuating according to the ordinary “ law of 
error" the " mean error," so defined, is about 1*25 times the 


average error, i.e. ^ each c being taken 

n 

positively, or in absolute quantity, as the phrase is, without 
respect to sign. 

The " frequency curve " of a series of magnitudes (such 
as a series of marks or errors in marking) is a curve of which 
the abscissae represent the magnitudes and the ordinates 
the comparative frequencies with which the corresponding 
magnitudes occur in the series. The frequency curves for 
errors in marking (and especially for errors on aggregates of 
marks, see § 17 below) appear in general to belong to the 
special type called the curve of error, of which Prof. Edge- 
worth likens the shape to the longitudinal section of a 
gendarme* s hat. 

(2) and (3) are derived from Prof. E.*s papers. 

^ For Summary of Paper II see p. 118 below. 
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a typical examination in which candidates are 
arranged according to the aggregate of marks which 
they have received on ten different papers, for each 
of which 300 marks are allotted, and assumes " that 
those who, without failing in any subject, have not 
obtained in all the subjects more than 2,000 marks, 
are placed in alphabetical order in the category of 
‘ Pass ’ ; and those who have obtained more than 
2,000 are placed in the category of Honours and 
arranged in' order of merit ’according to the numbei 
of marks received.” The paper investigates the 
error to which these aggregates of marks are liable ; 
and thence proceeds to deduce the degree of certainty 
incident to the distinctions grounded on these marks. 

No attempt is made to estimate the chances intro- 
duced by the possibility of a candidate being ‘‘ out 
of sorts ” on the day of examination, or of the paper 
being specially adapted to the accidents of his taste 
or reading; “ it is assumed, as in the case of our 
public examinations may be safely assumed, that the 
paper set in each case is an ideally fair one.” 

6 . Errors due to Minimum sensibile. — ^Prof. E. next 
points out that there must be a minimum sensibile 
in our perception of excellence, of which the magni- 
tude will vary with the subject, being least in Mathe- 
matics and most in subjects where marking depends 
greatly on ” taste.” 

Where subjects are for the most part of the simpler 
sort, Prof. E. estimates the ” probable error” of the 
mark for each answer at 5 per cent. ; in other words, 
when an examiner is putting down 20 (the maximum 
being larger) he very generally hesitates between 
19 and 21 as equally representing to him the worth 
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of the answer. Prof. E. then supposes that the 
average number of que^^stions attempted is 8o, and 
deduces that the '* probable error ” under this head 

is at least percent, or per cent, (see § 3above).‘ 

Thus if marks between 2,000 and 2,050 had been 
assigned to a candidate, there is an uncertainty on 
this account represented by a “ probable error ” of 
10. If the aggregate of marks follows what is often 
■<;alled the normal law (or frequency curve), this 
corresponds to a “ possible error ” of 20, the odds 
against such an error being only 6 to i, and a “ very 
improbable error ” of 40, against which the odds are 
250 to I. Even on the most limited view of the 
influence of chance a considerable uncertainty 
arises as to the true position of candidates in the 
Honours list. It is to be remembered that this 
error is altogetlier unavoidable. 

7 . Errors due to Personal Equation of Examiners. 
Variations of two kinds. — \ much more serious cause 
of inaccuracj* is the " personal equation ” of the 
examiners. Two species of variation are to be dis- 
tinguished under this head, (1) variations relating to 
particular answers, and (2) variations relating to the 
whole scale of marking. 

^ Thus in a pass examination in which pass or failure de- 
pends on the result of two papers, and say sixteen answers 

in all, the probable error would be — or 1*25 percent. 

I cannot help regretting that the author did not deal more 
completely with the simple case in which “ pass and 

failure '' depend on the results obtained in separate sub- 
jects, independently of the aggregate of marks obtained in 
all subjects. 
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Prof. E. states that his data (which are not quoted 
in full) show that the obseryed mean difference per 
cent, between two (numerous) sets of marks is 20. 
Taking into account an error of 5 per cent, due to the 
existence of a minimum sensibile, he finds the probable 
error due to the first cause to be 10 per cent. (“ corre- 
sponding to a just possible eri’or of 30 per cent.”). 
In the typical examination dealt with the probable 

error will be i-e. i‘i per cent.^ 

Prof. E. states that an experienced Oxford examiner 
tells him that, in general, where examiners are careful 
and experienced, ‘‘ marks run very close ; within (say) 
10 in translations (full marks being 100), within a 
class, or, better half a class, in subjects and essays 
(there being four classes, and the mark given being 
one of the letters a, y, 8, qualified in some cases 
by a -}- or a — to indicate a doubtful or intermediate 
position ”). 

Prof. E, gives, as an illustration of the second kind 
of variation, the difference of scale,‘a case of an 
examination in English in which he and a colleague 
“ had agreed beforehand as to the worth of the 
different questions, and as to the degree of proficiency 
to be expected of the candidates,” yet the average of 
his colleague’s marks came out 13 per cent, higher 
than his own, the former being 165 (out of a maximum 
of 300) the latter 145.“* Thus, if the examinations had 
been conducted on one occasion by the first examiner 

‘ If, as before, we take only sixteen answers, the probable 
error under this head will be 2'$ per cent. 

* The percentage refers to the mean of the two averages, 
viz. 155. (Footnote by Prof. E.) 
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and on another by the second, there would have been 
a difference of 13 per-cent, in the average marks 
allotted. 

Prof. E. supposes “ that in so well organised a 
system as that of our public examinations, care is 
taken to avoid this disturbance,” and that there is 
a *' tradition as to the relative value of each subject 
to be preserved by the slowly changing staff of chief 
examiners.” ^ 

« Still he says, ” in spite of conscious efforts to main- 
tain constancy, it would not be unreasonable, I think, 
to assume a probable error of 4 or 5 per cent, due to 
the factors under consideration. Thus, supposing 
there was no real difference between the papers in 
Mechanics of this year and the preceding, that the real 
average, so to speak, of both sets of papers was 150, 
yet in the sortition of examiners it is likely as not that 
the average may be pushed up to 156 or down to 
144 ; and it is possible that it may deviate much 
further from the ideal mean.” 

Prof. E. a^ds in a footnote that, in making this 
estimate, he stands astonished at his own moderation, 
” for there is no doubt that at many of our public 
examinations the fluctuations in the scale are enor- 
mous.” He quotes Dr. Venn, ‘‘an experienced 
examiner, who is also one of the highest living authori- 
ties on statistics,” as saying, ‘‘ I have frequently 
raised or depressed my own marks (or my colleagues’) 
by as much as 25 per cent, all through in order to 
bring them into general harmony.” 

‘ I presume that Ihrof. E. refers here to such examinations 
as Class I. Civil Service Examinations, where not only candi- 
dates, but ‘‘ subjects ” compete. 
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Adopting, however, the probable error of 4'5 per 
cent, in respect of personal, equation, the probable 

error on lo papers will be -^7^- or about i'5 percent.^ 

V 10 

An increase or diminution in the scale of marking 
in a given subject will not affect the relative order of 
candidates in that subject ; but it will obviously 
affect the relative order of the aggregates of marks 
obtained by candidates of whom only some take this 
subject, and therefore their relative order in the' 
Honours list.® 

Prof. E., by reasoning which he docs not give in 
full, comes to the conclusion that the additional 
probable error on the aggregates due to difference of 
scale is only from | to J per cent. " and may safely 
be neglected.”® 

8 . Errors due to Correction at great speed — or the 
" negligence of haste” — He next investigates the sub- 
ject of differences in marking due to difference of 
speed ; and finds that the minimum sensibile for papers 

%-5 

‘ If there are only two papers it will be or about 3 
per cent. 

® Prof. E. does not deal with the question of how far 
the pass in each subject is affected. Suppose the pass mark 
is 33 per cent,, the probable error on the 33 marks out of 
100 is 4. As I have said elsewhere. I find it difficult to attach 
any definite meaning to such a pass-mark as 33 per cent, 
on capacity tests, although it has a definite meaning in 
memory tests (see p. 63). It ought to be said that in many 
examinations, probably most, border-line cases receive 
special consideration which must reduce the probable error 
near the border-line, though it clearly cannot be eliminated, 
since the judgment of different examiners is bound to vary, 

» As will be seen from § 10 below. Prof. E. finally decided 
at a later stage not to neglect this error. 
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corrected at top speed becomes considerable. Taking 
this into account, he find^a probable error due to "neg- 
ligence of haste ” of about 25 per cent. If there are 
80 items affected by this error, the probable error of 

the aggregate is about 2 8 per cent. But the 

author assumes that the results in public examina- 
tions are, when Honours are concerned, subject to 
" comparatively leisurely revision and re-examina- 
^n by certain chief examiners,” and therefore does 
not take the item into account in dealing with the 
typical examination under consideration. He regards 
rapid examining as " safe enough for the purpose of 
a Pass examination.” 


9 . Errors dm to Fatigue of Examiners. — Prof. E. 
next investigates the error of marking due to 
the fatigue of examiners, which might be avoided 
if more time could be allowed. He thinks this may 
amount to from i to 2 per cent, on each paper; 
allowing i'5 per cent, and 10 papers, the probable 


error is 



say i per cent, on the aggregate 


marks for the examination. 


10 . Summary of Errors due to marking in Typical 
Examination. — For the typical examination Prof. E. 
sums up his results as follows ; — 

Percentage 
(probable) Error 
of Aggregate. 

Due to existence of a mtnimwm j 

Due to the examiner’s idiosyncrasies ; 

(1) as regards particular questions . • li 

(2) as regards scale . . . • li 

Due to the " negligence of examiners as 

regards scale ^ 


The total probable error expressed as percentage of 
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the aggregates of marks is the square root of the sum 
of the squares of the separate items, or rather more 
than 2 per cent. 

" For instance, if a candidate obtains an aggregate 
of 1,965 marks, thus falling below the Honour 
line by 35 marks, the odds that he ought by 
right to be in the Honours Class are .consider- 
able, say one in four. Even if he has obtained only 
1,800 marks, it is by no means certain that he is 
rightly placed among the rank and file of Pas^- 
candidates.” 

The same doubt exists as to the rightful position 
of those who are the same distance above the Honour 
line. “ To such an extent are the Honours shuffled 
by the fortune of examinations.” 

11 . Comparison of foregoing Results with certain 
Statistics of Cambridge Classical Tripos. — From 
an investigation of the marks of a Cambridge 
Classical Tripos examination, Prof. E. comes to 
the conclusion that the probable error on the 
mark of a single paper is not less than 9 per 
cent., and on the aggregate mark of 9 or 10 papers, 
3 per cent. 

Prof. E. gives an interesting table, too long to quote 
here, of the aggregates of marks for ten papers 
assigned to fifteen candidates by two examiners in 
each case. The greatest difference between two 
aggregates is 59, the two aggregates being 472 and 
413 ; the least difference is8,the two aggregates being 
632 and 624. Calling the difference between the two 
aggregates in each case and their mean the “ devia- 
tion,” we find the average deviation or “ error ” per 
cent, (of the mean marks in each case) to be 3'5. The 
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“ probable error ” is ‘84 of the average error, or 
about 3 per cent. (See also § 19 , p. 125). 

13 . Error due to possible non-representative Char- 
acter of Anm>ers of Candidates — Total Error of Marks 
in Typical Examination. — ^Prof. E. touches on the 
difficult problem of evaluating the error committed 
in taking the answers of the candidate as representa- 
tive of his proficiency (in other words, using the 
language of the text, what is the probability of the 
ssmple which he has given of his proficiency being a 
fair sample). He estimates the variation of marks 
from answer to answer to be adequately represented 
by a probable error of 25 or 30 per cent. ; and on this 
basis calculates the probable error on the aggregate 
of marks for typical examination to be about 2 per 
cent. 

Cumulating this error with the account already 
furnished, “ we find for the aggregate of marks, con- 
sidered as a test of the candidate's real proficiency, 
a probable error of about 3 per cent., corresponding 
to a possible Arorof more than 9 percent. It is to 
be remembered that this estimate leaves out of account 
many elements of uncertainty which are too subtle 
for the Calculus of Probabilities to handle ; such as 
the variation of the candidate’s spirits, or the circum- 
stance that the questions are not fairly representa- 
tive of the subject. ... It is to be remembered, 
also, that the error in appreciating the answers 
is reckoned from the supposed ideal mean which 
would be presented if a large number of com- 
petent sets of critics were to examine the work. 
If it is asked what difierence is to be expected 
between the results of any two sets of examiners, 
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the estimate of probable error must be somewhat 
increased,” , 

13 . In how many Cases may award, of Honours in 
Typical Examination be due to Chance ? — Prof. E. next 
touches on the question (i) in how many cases 
the obtaining or not obtaining Honours may be re- 
garded as accidental, and for this purpose makes the 
assumption, which he thinks legitimate, that out of 
every i,ooo candidates, there are among those who 
have passed, at least 120 who have obtained mai;ks 
between 1,800 and 2,000, On this basis he calculates 
that the average number of candidates who are placed 
in the wrong category owing to chance is about 30, 
although in one year it might be only 20 and in another 
rise to 40, 

14 . Recommendations for Reform: (i) Honours Lists. 
(2) Working of Assistant Examiners. (3) Question of 
making Examination-marks conform to a given Fre- 
quency Curve. — To mitigate the defects arising neces- 
sarily from chance, Prof. E. makes certain recommen- 
dations " based on purely statistical reasons,” which 
he is quite prepared to find may be overborne for 
purely practical reasons. 

(i) Arrangements of Honours Lists. — He suggests 
that '' the arrangement of an Honours List in an 
ostensible order of merit should be abandoned when 
the distinctions thus drawn do not correspond with 
any degree of probability to real differences. The 
extent of each homogeneous alphabetically arranged 
class should be considerably greater than the probable 
error incident to the aggregate of marks upon which 
the arrangement is based. Thus, in the example 
chosen, if an aggregate of marks between 2,000 and 
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2,100 is liable to a probable error of 50, then all or 
most of the candidates Ipetween these limits might be 
placed in one homogeneous class.* 

(2) Question of Assistant Examiners . — He suggests 
that the functions of Assistant Examiners requires 
special consideration, and says there appear to be 
two distinct theories, {a) and {b), with regard to the 
arrangement in respect of Honours candidates who 
[are reported by the assistant examiners to] have 
Jessed ; and that the two theories may admit of 
intermediate shades and practical compromises. 

{a) According to theory (a) the function of assis- 
tants is merely to select provisionally, for careful re- 
e.xamination ab initio by the chief examiners, candi- 
dates who may be deserving of Honours. 

In this case, the papers selected for re- examination 
should be defined, not by the Honour line of 2,000 
marks, but by one considerably below it, due account 
being taken of the ex hypothesi rough marking of the 
assistants being affected by a very large probable 
error. Prof. £. thinks that, according to this view, 
in order to avoid mistakes it would be necessary for 
the chief examiners to re-examine completely and 
de novo the papers of about half the candidates who 
have passed. 

* Professor Arthur Schuster, on the other hand (Univtr- 
sily Review, May 1905), has pointed out that the differentia- 
tion into classes without any indication of order of merit, 
may make an unjust distinction between the last in one class 
and the first in the next. The creation of a large number 
of classes in a typical examination such as that postulated 
by Prof. E., with a class in alphabetical order for each multiple 
of 100 marks, might be a satisfactory compromise ; it would 
not be necessary to make use of the word class, the candi- 
dates within each century would merely be bracketed. 
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(ft) According to this theory, the marks assigned by 
the assistant examiner "are, final, subject to the in- 
spection of the chief examiner, and the revision of ex- 
ceptional cases.” According to this principle, " or any 
intermediate plan in so far as it involves this principle, 
it maybe expected that the sources of error which have 
been above described under the head of negligence * 
will become peculiarly operative.” 

Prof. E. at once mitigates this criticism by a foot- 
note, in which he says ; “ What maybe safely affirme(t 
of any public examination is, on the one hand, that 
any method which has commended itself to many 
able and experienced examiners is fairly good ; but 
that, on the other hand, it is not likely to be the 
best possible, if it has been adopted without reference 
to the theorems in Probabilities which evidently 
dominate the whole subject.” 

In order to avoid the error due to marking at high 
speed. Prof. E. next investigates the possibility of 
combining rapid dealing and accuracy by omitting 
the correction of half the answers ini each subject 
and comes to the conclusion that the probable error 
incident to the aggregate of marks obtained by 
doubling the marks so allotted (the correction being 
leisurely) is only i’4 times the probable error (and 
not twice the probable error) of the aggregate obtained 

1 I think the word " neghgence/' though used by Prof. E., 
implies a moral stricture on the individual examiner or 
assistant concerned which is not intended by the author, who 
points out, in dealing with this matter, that "the number 
of the staff in relation to the quantity of work to be done, 
and the time for doing it, is sometimes such as to necessi- 
tate an extraordinarily rapid rate of examining/' Prof. E. 
tells me he concurs. 
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by correcting all the answers, the correction being 
leisurely. , 

Thus if “ the total aggregate of marks, considered 
as representing the real proficiency of the candidate 
is liable to an error of 2 or 3 per cent, [see § 10, above] 
the error incident to an examination of half the 
answers is only 3'5 or 4'5 , the number of answers being 
large, and the half selected at random.” 

Prof. E. suggests, as a simple method of diminish- 
ing the time taken over correction which would avoid 
hasty correction is to increase the number of assist- 
ant examiners, and further that the answers of the 
candidates should be separable, the answers to (say) 
the first two questions of all the candidates being 
looked over by examiner A, the answers to the second 
two by examiner B, and so on. ” By this arrange- 
ment the influence of the personal equation, intro- 
duced by a plurality of examiners,” would be so dis- 
tributed as to become insensible.^ 

( 3 ) Qiicstion of making Examination-marks conform 
• 

‘ This arrangement is adopted in certain examinations. 
It has the further advantage that the examiner, who neces- 
sarily marks slowly at first, and increases speed as he becomes 
more familiar with the difficulties of the paper and the 
problems involved, attains the ma.ximum speed consistent 
with care more quickly than if he has to deal with, say, four 
or five times as many different questions. The disadvantage 
is that he cannot judge of any candidate’s work as a whole. 
It sometimes appears from the correction of a whole paper, 
in a way that correction of a part would not reveal, that 
the candidate is making correct use of terms which he does 
not in the least understand. Hence it would appear, as Prof, 
E. points out, that the principle would require '* to be tempered 
with practical sagacity and knowledge of details” before it 
could be immediately applicable. 
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io a given Frequency Curve. — Prof. E. suggests that it 
may be possible, in each sub.iect, to determine what 
ought to be the proper average for a set of marks of 
different candidates at a given examination and the 
proper " dispersion ” of the marks about this average ; 
and he shows how another set of marks may be 
reduced so as to conform to this arrangement. The 
suggestion is repeated in the second paper in respect 
of Government examinations (see § 25 below).^ 

Paper II 

16. The Differences of marking between different 
Examiners, in respect of one Paper, are similar to the 
corresponding Differences in respect of another Paper 
in the same Subject. — Comparison in the case of different 
subjects (Geometry and History ). — After certain pre- 
liminary matter (see § 1 above) the author discusses 
the general question of laws of error attaching to 
different kinds of judgments or estimates, relating 

» In revising this abstract, Prof. E. drcwvs attention to 
the great importance of paying attention to " dispersion " 
when marks are combined, and points out that if the award 
of a prize is decided by the sum of marks obtained in Latin 
and Greek, and if, while the average obtained in the two 
subjects is the same, the " dispersion ” is much greater in 
Greek, the Latin marks will tend to become inoperative. He 
illustrates the argument by curves which cannot be repro- 
duced here; but it is easy to see that if the marks of, say, 
the first half-dozen candidates lie close together in Latin, 
but are widely dispersed in Greek, the Greek marks would 
decide the matter. This result would be unfair if it were 
due only to the difference of idiosyncrasy of the Latin ex- 
aminers and the Greek examiners in their method of assigning 
marks. In such a case verbal discussion of the results is 
essential for a fair award. 
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to things other than examination answers, e.g. 
weights and heights. He discusses the case of 
marks awarded by sixteen different competent 
examiners to the work in mathematics of a pupil 
in the sixth form of the North London Collegiate 
School, and finds the probable error deduced 
from the sixteen observations to be 4 5 per cent. 
" For the eight first in the order of entry it is 3 5 ; 
for the second batch it is 4*5. Accordingly, it is not 
a^leap in the dark when we infer that what is true 
of one set of sixteen examiners marking that paper 
would be approximately true of another set of 
equally competent examiners,” He finds, further, 
that the law of dispersion for two different papers 
corrected by the same examiners is also approximately 
the same. Although he expected the variation in 
the markmg of history papers to be greater than in 
the case of mathematics, the limited experience 
afforded by three sets of marks assigned by nine 
different examiners to three history papers showed no 
material diff^cnce between the two cases. 

16 . Marking of piece of Latin Composition hy twenty- 
eight different Examiners. 

I think it well to quote the following passage 
verbatim: — "The experiment which I shall next 
adduce is in alia materia. It relates to a piece of 
I.atin prose which the Editor of the Journal of 
Education kindly allowed me to insert in a recent 
number of that periodical, with a preface which it 
may be well here to reproduce in part ; — 

‘ I propose, through the medium of the Journal 
of Education, to invite any competent person to 
assign a mark to the subjoined piece of Latin 
119 



EXAMINATIONS— NOTE E 


prose, upon the supposition that he is marking 
the work of a candidate for the India Civil 
Service. Let it be disfinctly understood that 
in giving his mark the examiner is not to 
look to, or wish to illustrate, his own ideal of 
classical elegance, nor yet the degree of pro- 
ficiency which may be current in the school or 
other institution with which he may be con- 
nected. Let him imagine that he has been 
appointed examiner in Latin for the India 
Civil Service, and let him give his marie 
having regard only to what may be expected 
from a candidate for that prize. Let loo be 
the maximum attainable by any candidate. 

“ ‘ To avoid accidental divergence as much as 
possible, to perform the experiment under the 
most favourable conditions, I would suggest that 
the examiners should consist of a pretty homo- 
geneous class — of much the same class as those 
who actually conduct our public examinations. 
To be more definite, I would invit^ to take part 
in this experiment only those who have taken 
high honours in classics at one of the univer- 
sities, or classical masters of the sixth form in a 
public school. All such are earnestly invited 
to examine the accompanying piece with as much 
care as if they really were exercising the function 
of a public examiner ; and to send to the editor 
their verdict, guaranteed by their name and 
status, which, it need hardly be added, it is not 
intended to publish. It is desirable that the 
examiners should assign their respective marks 
independently, and without mutual conference.' 
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“In response to this appeal twenty-eight highly 
competent examiners were so kind as to mark the 
piece of Latin prose. The grouping of these twenty- 
eight independent marks may be expressed by the 
accompanying scheme of figures, with the interpre- 
tation of which the reader will by this time be 
familiar ‘ : 

II V VII VII V II 

635 735 785 833 935 

* “The actual marks are: — 45, 59, 67, 67'5, 70, 70, 
72-5. 75 . 75 . 75 . 75 . 75 . 75 . 77 . 80, 80, 80, 80, 80, 82, 
82, 85, 85, 87-5, 88, 90, 100, 100. 

“ It will be seen that 78^5 is the central value, the 
presumably correct mark * ; the probable error on 
either side of the correct mark is 5. It may excite 
surprise that this error is not very much greater 
than what we observed in the case of the geometry 
and history, which, without offence, maybe described 
as not requiring on the part of the examiner such 
an exercise of literary taste as the Latin prose. But 
when we consider the extreme as well as the central 
portions of the Latin prose group, we shall be justified 
in attributing to the marking of the more advanced 
work a larger element of chance. It is interesting 
to observe that the two highest marks, 100, are (each) 
more than double the lowest mark, 45. An occur- 
rence of this kind is not at all uncommon in the 

• The Roman numerals express the number of awards of 
marks of which the amounts lie between the Arabic figures 
on either side of them. Thus there were two marks below 
63’5 ; five between 63’5 and 73 5 ; seven between 73 5 and 
78'5, and so on. 

* The arithmetic mean is 78 (Prof. E.’s footnote). 
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marking of advanced work, where there is room for 
great diversity of taste in the examiners/’^ 

16. Prof. E. deduces (with* the help of the results 
set forth in § 8 above) that '‘in an examination of 
one subject admitting as much diversity of judgment 
as the Latin prose ... a difference of 25 per cent, 
between the marks of two candidates may be acci- 
dental and liable to reversal. Suppose, for instance, 
that, under one examiner, candidate A. obtains 90, 
and candidate B. 70, that is, a difference of 25 p§r 
cent, of the average between the two marks, namely 

* No doubt the extremer divergencies are partly due to a 
dificrence, not of taste, but of scale. In the case of English 
composition and the speculative sciences, I have met with 
equally great discrepancies in the marks given by two 
examiners to a particular candidate, without any corre- 
sponding difference in the average of their marks given to a 
whole set of candidates. (Footnote by Prof. E.) 

Prof. E. now adds the following interesting note in re- 
gard to this experiment : — " As to the Latin prose, I ought 
perhaps to explain that in one respect it was hardly typical. 
Having written it as well as I could, I deliberately dashed it 
with an elementary error, the use of the preposition penes 
with the which, as far as I know, never occurs. This 

schoolboy error, no doubt, as I intended, utterly puzzled 
the judges. One broad-minded man, now the Provost of a 
distinguished College, said that such a mistake in such a 
paper might be treated as a mere lapsus pluma, and he 
gave it full marks. Another, I think the one who marked 
45, thought the mistake was unpardonable.*' 

According to my own experience. Professor E.'s inten- 
tional mistake increases rather than diminishes the value of 
his experiment. A friend of mine who was proxime accessit 
for either the Hertford or Ireland, was told by the examiners 
that except for one thing he would have been awarded the 
prize ; he had made one unpardonable mistake in a quan- 
tity in a Latin verse. 
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8o ... it is not a very improbable event — say about 
as unlikely as throwing heads at pitch-farthing six 
times running [say i to loo] — that under another 
examiner the order of the candidates might be re- 
versed, B coming out above, though probably not high 
above, his rival.” 

17 . Error of Aggregate of Marks formed by Addition 
of 'l otah for different Subjects. — It is necessary to 
deal with aggregates of marks formed by the addition 
0 ^ totals for different subjects for which the laws 
of error are different ; but Professor E. states that 
'* When a compound mark is formed by the addition 
of several marks on different subjects, the manner 
of divergence, or law of error, appertaining to the 
compound is of one and the same general type, 
whatever the type of the particular laws appertain- 
ing to the several components. For instance, let 
the several subjects be geometry, history, Latin prose, 
etc., and let the law of error proper to each of these 
subjects be obtained by observations such as those 
which have been adduced in the preceding pages. 
Then the compound (mark in geometry -f mark in 
history -f mark in Latin prose -f etc.) will vary with 
the change of examiners according to a law of error 
not in general identical with that of the components, 
but of a certain normal universal type which may be, 
and is often, called the law of error.” 

Prof. E. states that what is true of sums holds 
good for differences, and this result is used in the 
last paragraph of § 16 above. 

18 . General Case of Distinction awarded for Aggre- 
gates of Marks in different Subjects — Use of Results 
obtained from two Markings of the same Work by the 
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Examiner at different Times. — Prof. E. now deals 
with the general case in which distinction is deter- 
mined by the sum of marks in different subjects. 

He suggests that if we had to depend on data 
yielded by marking of the same work by several 
examiners, we might despair of obtaining sufficient 
statistical material for the reasoning, but thinks it 
suffices to combine a few instances of this type with 
a great many instances of two examiners both mark- 
ing several pieces of work. 

Prof. E. investigated the results of examination by 
the same examiner (Mrs. Bryant, D.Sc.) of the same 
forty papers written by her pupils, on two occasions 
separated by a considerable interval of time. He 
does not give the result directly but states that if 
the same candidate were examined in ten papers by 
two sets of ten examiners, and if each couple of 
examiners “ marked on the whole at the same level 
but with respect to particular papers disagreed with 
each other about as much as Mrs. Bryant with her- 
self, then the difference between their»marks would 
in the long run as often as not amount to 2 5 per cent, 
(of the average mark for the whole set of ten papers 
examined).” 

Thus, if a candidate obtains 2 '5 per cent, of the 
average mark for the whole set of candidates above 
the mark for the Honours line, it is as likely as not 
that under another set of examiners he would have 
come out either at least as high as five per cent, above 
the line, or at least as low as the line. It is with the 
latter alternative only that we are concerned. The 

! Further details are given in the paper by Prof, E. in the 
Philosophical Magazine for August, 1890. 
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probability of this is only half a half, that is, a 
quarter.” ^ , 

If a man clears the Honours line by lo per cent, 
of the average marks, the odds are 300 to i against 
a reversal by a different set of examiners ; if he clears 
them by 9 5 per cent., the odds are 200 to i against 
reversal ; if he clears them by 7 per cent, they are 
only 50 to I. 

In the case of the award of appointments it is, as 
a jrule, not an Honours line that is fixed in advance, 
but the number of appointments awarded. Prof. E. 
assumes that the difference of marking between 
examiners will be not greater than that between the 
two markings by Mrs. Bryant of the same papers 
on different occasions, and applies this to the case 
of an examination for fifty clerkships in 1875, for 
which there were 172 candidates. He comes to the 
conclusion that there are only 17 candidates for 
whom the odds against being displaced if there were 
a change of examiners is as great as 50 to i. 

19 . Details in regard to a Classical T ripos Examina- 
tion. — Prof. E. discusses a series of three sets of 
marks given by three examiners to 75 papers in com- 
position. The average of two of the examiners was 
about the same, but the average of the third examiner 
was about ^ less than this and was increased there- 
fore by 20 per cent, for purposes of comparison. 

^ In other words it is equal odds whether he is within 
the limit of 5 per cent, above Honours, or outside; it is 
4 to I against his being outside the limits below the lower 
line, and it is 4 to i against his being outside them above 
the upper line. If a large number of cases were taken, one 
half would fall within the limits, one quarter below them, 
and one quarter above them. 

125 



EXAMINATIONS— NOTE E 

The coefficient of divergence might then be expressed 
as a “ probable error of 5 pej; cent, on the sum of 10 
marks.” 

" The severer examiner is as often as not distanced 
by one of his colleagues to such an extent that one 
mark is double the other. Fourteen times out of 
ninety, one mark is three times as great as another 
given to the same work. Four times, the excess is 
in the ratio 5 to i. And once the severer examiner 
exceeds his more lenient colleague in the ratio of i6»to 
I. These graver discrepancies may no doubt be ob- 
viated by properly correcting the scale of the severer 
examiner. Still the effect of the difference in scale, 
and the efficiency of correction, may easily be exag- 
gerated. It may surprise even experts to hear that, 
as between two examiners, A and B, who upon the 
whole are fairly well abreast, occasionally A is ahead 
of B, and B ahead of A to the extent of 50 per cent.” 
(See also § 11.) 

20 . Details in regard to Oxford Liferce Humaniores 
Examination in Philosophy. — Prof. ’E. finds the 
uncertainty in the marking at the Cambridge Classical 
Tripos to be about the same as the uncertainty in the 
marking in Philosophy at the Oxford Final Honours 
in Litera Humaniores, of which he gives some details. 
He points out that the fluctuation in the estimates of 
those who are in or near the first class is markedly 
less than for le.ss successful candidates. "In fact,” 
he says, “it is as small as any which I have ob- 
served in the case of subjects more elementary and 
precise ” ; a result which he regards as a striking 
tribute to the Oxford method of marking. 

21 . Details of Examinations in English Composition 
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and Speculative Science. — Prof. E. next investigates 
the marks allotted by two different examiners to 400 
pieces of English composition. The "mean error” 
{i.e. the square root of the sum of the squares of the 
400 differences between the individual pairs of marks, 
divided by y^oo) is 67 marks.^ If, instead of taking 
the whole 400 observations, they are divided into 
three batches of 133, the corresponding figures are 
63, 67, and 72. 

,The author next points out that there was a differ- 
ence of " scale ” between the two sets of marks 
amounting to 10 per cent, of the average mark. Re- 
ducing the marks to the same scale the " mean error ” 
is only reduced to 64 marks. 

The average of the 800 marks for the whole set of 
papers then becomes 227 ; and the " mean error ” 64 
is about 28 per cent, of this average. If there were 
ten papers of this kind, the “ mean error ” for the 
28 

ten would be per cent., and the “probable 

error ” would be 6 per cent. 

Thus, on the aggregate marks for the papers for 
each of which there was a " mean error ” of 28 per 
cent,, the error as likely as not to occur would be 
6j)er cent, of the average mark. In a public examina- 
tion on speculative science “ not guarded so carefully 
as the Oxford Examination in Philosophy,” the 
probable error for a set of ten papers amounted to 
8 per cent. 

22. Summary of Previous Results . — ^The author now 
sums up his results in the following table of the dis- 
crepancies as likely as not to occur between the sums 

* Cf. footnote on pp. 104*5 above. 
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of marks on ten papers assigned by two sets of ten 
examiners to the same papers. " The reader will 
recollect that each of these coefficients requires to be 
multiplied by at least 3*5 or 4 in order to obtain the 
height above the Honour line at which a candidate's 
position may be regarded as irreversible.” 


Number of Probable error calculated for 
papers a set of 10 such papers, ex- 
scrutinised. pressed as percentage of the 
average aggregate mark. 

High School. 6th Form, 

Geometry. (Marked twice 
by the same examiner at 


different times [see § 18 ]) 
High School, 6th Form, 

82 

2’5 

Geometry and History 
Cambridge Classical Tripos, 
Composition and transla- 
tion, History and Grammar 

160 

3*0 

(mixed) (see § 19 ) . 

Latin prose of the type set at 
the I.C.S. Examination 

480 

4 

(see § 16 ) 

Oxford Literae Humaniores, 

28 

4’5 

Philosophy only (see § 20 ) 
Cambridge Classical Tripos, 

460 

4'5 

Composition only (see § 19 ) 
English Composition (see § 

2^5 

3 

21) .... 

800 

6 

Speculative Science (see § 21 ) 

140 

8 


23 . Question of Measure for Element of Chance in any 
Public Examination. — Prof. E., ” inclining to the side 
of moderation,” adopts as the probable error for marks 
at public examinations the lowest based on the com- 
parison of results of different examiners, viz. 3 per 
c^nt. of the average mark obtained at the examina- 
tion. 
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“ Adopting this datum,” he writes, " I am able 
to answer the following question with respect 
to any examination of which the marks are pub- 
lished : How many of the candidates placed are 
quite safe, in this sense, that, if their work had 
been marked by another equally competent set 
of examiners, the failure of any assigned one of 
the successful is a very unlikely event, as un- 
likely as getting a run of six or seven heads or 
tails at pitch-farthing ? » The public examina- 
tions in which I have sought the answer to this 
question fall into three categories; the India 
Civil Service, the Army, and the Home Civil 
Service Clerkships of the second order. I find 
the element of chance in these public examina- 
tions to be such that only a fraction — from a 
third to two-thirds — of the successful candidates 
can be regarded as quite safe, above the danger 
of coming out unsuccessful if a set of equally 
competent examiners happened to be appointed. 
A corresponding proportion of the successful — 
from two-thirds to one-third — ^must be described 
as unsafe. A rather larger number — though 
much smaller proportion — of the unsuccessful 
candidates would have a chance of succeeding at 
a re-examination ; for instance, at three Clerk- 
ships Examinations, 25, 43, 72, out of a total 
number of unsuccessful, respectively, 126, 102, 
and 121. 

“This question may also be asked: What 

* Odds of 100 to I, corresponding to a height above the 
Honour line of 3'5 times the " probable discrepancy.” (Foot- 
note by Prof. E.) * 

K 
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number of displacements is most likely to occur 
in the event of a re-examination ? The answers 
in some cases which I have scrutinised proves to 
be about a seventh part of the successful. Thus, 
where fifty clerks were appointed, it may be 
expected — ^the most probable event is — that seven 
would be displaced by a re-examination of the 
same work.” 

Prof. E. gives here an elaborate table relating.to 
the results of a number of Civil Service and Army 
Examinations for which reference must be made to 
the original. 

Prof. E. comes to the conclusion that the distinc- 
tion between adjacent candidates at such examina- 
tions as arranged in accordance with the marks 
allotted is in hardly any case a real distinction. 
” Differences of marks on the strength of which one 
Civil Servant is promoted ‘ above his fellow correspond 
to only the faintest probability of real superiority. 
Occasionally, but by no means invariably — in exa- 
minations decided by an aggregate of marks — the 
senior or the first two or three break away from their 
neighbours by an interval which cannot be ascribed 
to accident. But between the adjacent men among 
the rank and file the distinction is hardly ever real.” 

24. Comments by Prof. E. on his Results. — ^Prof. E. 
thinks his results err, if anything, on the side of 
moderation, for the following reasons: (i) The cor- 
rection for mutually compensating errors (see 

I The author, by " promoted," means here " placed at 
the entrance examination " above another candidate, being 
thus entitled to a superior initial position in the service. 
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§ 3) is probably an over-correction, and there is " an 
intractable minimum of constant error ” ; (2) the 
division used for this correction, a/io. is too great 
for some examinations, the average number of papers 
being probably nearer seven than ten ; (3) the assump- 
tion that differences of scale of marking (see § 7) are 
corrected, or can be allowed for is “ precarious,” 
since candidates do not always take the same sub- 
jects ; (4) owing to changes of scale, the hypothesis 
that the Honour line may be regarded as constant, 
whatever the change of examiners, does not hold. 
Hence the error is greater than that shown. 

On the other hand, the uncertainty about high 
marks appears to be less than about lower ones ; 
in order to obtain the best results we should consider 
only those above the line of what may be regarded as 
certain failure.^ “ Possibly the uncertainty of some 
particular examinations may have been exaggerated.” 

26. Question of making M arks of a given Examination 
conform to a given Frequency Curve [see § 15 (3)]. — 
Prof. E. suggests that any hard-and-fast rule would be 
dangerous. Referring presumably to Government 
examinations, he says; 

“ Perhaps the best general principle is that 
the ' quartile ’ and ‘ median ’ should be fixed for 

^ It seems clear a priori that examiners will not take as 
much trouble in regard to candidates whom they regard as 
unlikely to succeed. I do not know whether, at the date 
of the examinations investigated by Prof. E., the Civil Service 
Examiners made the deduction for smattering of which 
the existence has long been known, but which was first 
officially described in the Report of Mr. Stanley LeatJ^es' 
Committee (see p. 17), 
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each class of examination by a Committee of 
permanent officials, in the light of an extensive 
experience in pari materia ; that the examiner 
on each occasion should be expected to 
furnish a set of marks roughly conforming to 
the pattern thus prescribed ; and that any 
adjustment which the marks furnished may seem 
to require, should not be performed without the 
concurrence of the acting examiner, who has scru- 
tinised the merit of each particular candidate,” 

26. Inevitability of an Element of Chance — Sugges- 
tions for the Future . — Professor E. points out that there 
is an incorrigible minimum of uncertainty, and is 
satisfied that in some of the cases investigated that 
minimum could not be substantially reduced by any 
method of correction. 

For hardship entailed on unsuccessful candidates 
excluded by chance he says : “ Perhaps here, as in 
other grievances, mere publicity is a sufficient, and the 
sole obtainable, remedy. The general recognition of 
the element of chance in examinations would miti- 
gate the disappointment of those who, for want of a 
few marks, are either altogether excluded from the 
class of the successful, or compelled without prefer- 
ment to take a low place in that class.” * 

Prof. E. suggests, rather for the sake of facilitating 
the conception of his theory ” than as a serious prac- 
tical proposal,” that candidates within the limits of 
uncertainty should draw lots for the vacant places 
allotable, each candidate receiving a number of tickets 
proportional to the chance that he is really deserving, 
as accurately ascertained by the theory of proba- 
bilities. He points out that sucl/ a proposal would 
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initiate the sense of injustice in disappointed indi- 
viduals and " do more to bring home to the public 
mind the aleatory character of examinations than 
even an article in the Journal of the Royal Statistical 
Society.” 

Prof. E. points out the element of certainty as well 
as that of chance in public examinations. He com- 
pares their results to those of a barometer uncorrected 
for violent changes of temperature. " That examina- 
tions are a very rough test of merit, but they do serve 
to distinguish excellence in certain qualities, is a con- 
clusion which recommends itself to practical and 
sensible persons.” 

But ” mere common sense, without the calculus of 
probabilities, is not competent adequately to define 
the limits of chance and certainty. Common sense 
might be embarrassed in defending its position against 
the awkward fact which has been adduced, that even 
in the best regulated examinations one examiner 
occasionally differs from another to the extent of 50 
per cent. ; for common sense cannot fully appreciate 
that mutual compensation of errors in virtue of 
which a peculiar stability attaches to aggregates of 
marks.” 

, In a charming peroration (so suave one hardly dare 
suspect it of irony) Prof. E. suggests that ” To 
examiners at least it wUl be interesting to test the 
accuracy of the instrument with which they work. 
The statistical study may beguile the monotony of 
their task.” 

27. Conclusion . — In presenting the above summary 
of Professor Edgeworth's papers I must disclaim once 
more any pretension to be an authority on statistics 
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or to add by quotation any weight to Prof. Edge- 
worth’s results. They are tho^e of a recognised ex- 
pert to whom one cannot but be grateful. In spite 
of the soothing remarks which I have, I hope, repro- 
duced as faithfully as the disturbing figures, the 
papers leave me, and I believe they will leave others, 
with the conviction that the necessity for further 
investigation is real and pressing. Prof. Edgeworth’s 
results, significant as they are, are based on data 
some thirty years old. What are the facts now 2 
Ought not each examination on a large scale to be 
periodically subjected to the criticism of a professional 
statistician ? We cannot avoid the effects of chance 
in examinations — ^such chance exists in all human 
affairs — ^but it is surely a duty both to ascertain what 
they are, and to reduce them to a minimum. 

And I repeat once more my belief that, if examiners 
asked themselves exactly how they could define in 
words what the passing of an examination means, 
so as to be able to give a definite assurance to the 
public in regard to what all the successful candidates 
can do, the element of chance would be singu- 
larly reduced. 

I make one last quotation from Prof. Edgeworth, 
which is appropriate here though he uses it in a some-, 
what different connection : 

" What should we think of a company who, 
dealing in thousands and millions sterling, 
grudged the payment of a professional account- 
ant to check the accounts of their Directors ? 
It would be a similarly false economy on the part 
of educated society, considering the stupendous 
^sum of mental^labour involved^in the system 
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of competitive examinations, to grudge the ex- 
penditure of a little additional effort on a tech- 
nique which profe^es to bring to book, and test 
the figures of the examining bodies who are 
responsible for the direction of that labour.” 
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ON KNOWLEDGE-TESTS AND MEMORY-TESTS 

It is frequently impossible in a written examination 
to distinguish knowledge, * implying understanding, 
from pure memory. The instance of the candidate 
who, in translating a passage from a prepared book, 
carries his translation beyond the passage set, illus- 
trates the point. 

It has been suggested by a critic that I have over- 
estimated the importance attached to memory-tests 
in examinations at the present time. I may point 
out in this connection that Prof. Burnet, in his Higher 
Education and the War, strongly advocates the use 
of set books, and deprecates the use of " unseen trans- 
lation ” in classical teaching in schools. I am not 
prepared to contest the soundness of his views in this 
particular matter ; I quote them as showing that 
memory-tests are not obsolete, and that precautions 
should be taken in using them. 

Since the publication of the above criticism, on§ 
extremely successful candidate told me that, having 
to pass examinations in English (which, although he 
is a British subject, is not his mother-tongue) he learnt 
off by heart answers which he did not understand to 
questions which he recognised by catch-words, and 
passed a series of examinations in this way. He 
assured me that his case was an extremely common 
one^in his own part of the British dominions. An 
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Englishwoman who was first or second in each of the 
four subjects of a ^rst-year University course in 
natural science at one of the largest University 
colleges in Great Britain a few years since, has also 
assured me that in physics, in which she took the 
second place, she would have been quite incapable 
of explaining some of her own answers ; and that her 
personal success gave her a rooted disbelief in the 
results of examinations as now conducted. 

. Charles Lister, while serving his diplomatic appren- 
ticeship at Rome in 1912, commented on “ the bliss- 
ful ignorance of commerce ” in the diplomatic service 
(now about to be remedied), and wrote: “ I should 
suggest some compulsory subject in the examina- 
tions in banking, finance, and trade, but probably 
some Hawkins would come along and put it all into 
a book in the form most easily learnt and most easily 
forgotten, and little would come of it ” {Charles 
Lister : Letters and Recollections, 1917, p. 45). In the 
considerable literature on examinations the views of 
the recently examined are comparatively rare ; I 
attach importance to them, especially in the case of 
able candidates such as those above cited. My 
critic, M. H. (see Times Educational Supplement 
for August 2, 16, and 23, 1917) writes : “ Only after- 
life can show how far the acquirements of memory 
are fleeting : unconscious stores may play their part 
in the intellect ; it is better to have learned and 
lost than never to have learned.” The excellent 
epigram seems to me applicable only to knowledge 
which has been assimilated. The examples quoted 
above show that unassimilated knowledge may and 
does still serve its turn in the examination- roonv 

137 



NOTE G 


ON THE USE OF THE TERM "ORDER 
OF MERIT” 

I HAVE consulted my friend Prof. S. Alexander, on 
the significance attached to the word " merit,” 
as used by professional writers on ethics ; he has 
kindly referred me to Leslie Stephen’s Science of 
Ethics, ch. vii., § i Merit, and § 2 Effort ; James 
Martineau’s Types of Ethical Theory, Vol. IL, Book I, 
ch. iii.. Merit and Demerit ; and to his own Moral 
Order and Progress. In the last-named work (p. 194), 
Prof. Alexander writes of merit ; " Though in reality 
a simple idea, it has given rise to the greatest con- 
troversy, from the apparent divergence with which 
it is applied.” This is not the place to discuss 
general theories of ethics, even if the writer were 
competent to do so. But the objection to the use 
of the word " merit ” m the phrase " order of merit 'i 
is that it really leads to confusion of ideas, as indi- 
cated in § 17 , p. 63 above. I find myself in modest 
disagreement with Prof. Alexander's reconciliation of 
different views, and in general sympathy with Mar- 
tineau’s interpretation, which Prof. Alexander thinks 
is also that of " the man in the street." 

I think that confusion arises frequently because 
in using the word one is apt, when speaking of an 
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action, to lose the distinction between {a) the intrinsic 
merit of the action considered independently of the 
author, and {b) the merit involved in its performance. 
The more dazzling the action, the more difficult does 
the public find it to make the distinction. A chance 
observation of scientific importance may confer un- 
dying fame on the observer. Heroism may confer 
immortality on a failure. 

But the reality of the distinction will be made 
clear by a concrete hypothetical example connected 
with our subject. Suppose you are shown a short 
piece of French prose, irreproachable in grammar and 
" quite good ” in style, of which you do not know 
the author. You say, “This piece has merit.” You 
then ask by whom it was written, and are told that it 
was written (say) by Anatole France. “ Ah,” you 
say, " not up to his usual level.” You are then 
informed that the previous information was mistaken, 
and that the piece was by an English boy who has 
never been in France ; and you are filled with admira- 
tion. Clearly, one attributes great merit to the 
English schoolboy for a performance for which one 
would attribute none to Anatole France, yet the 
value of the piece of prose itself remains the same. 

•The example is not so far removed from reality as 
it may seem. While it is quite true that French 
authors do not enter for our examinations in French, 
candidates who have been partly trained m France 
do so, and compete against others who have con- 
tended against the greatest possible difficulties, it 
may be both of natural disposition and of external 
opportunity, in order to achieve the same result 
as those more favoured by nature and by fortune, 

139 



EXAMIN ATIONS-NOTE G 

Clearly we have no right to say, then, that examina- 
tions place the candidates t\(!mselves in " order of 
merit.” But we ought to be justified in saying that 
examiners place candidates in the order of proficiency 
shown by their answers, which is another matter. 

To the philosophical critic, I venture to point out 
that the case is simplified by the fact that exami- 
nation answers are not (in general) productions so 
dazzling that all distinction between the merit of the 
author and of the production is perforce obliterated 
in the public eye and understanding. Their intrinsic 
public value, except in a few delightful instances, is 
nought. Having served their purpose, they pass, 
unregretted, to the oblivion of the paper-mill. 
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